In manufacturing engineering an important task is to assess conformance of a manufactured part to its design specification. This task is usually carried out by measuring the manufactured part using a coordinate measuring machine to provide a set of coordinate data. Software is used to fit the design surface to the data and the closeness of the data to the fitted surface is examined to determine if the manufactured part is within tolerance. The result of this process strongly depends on the correctness of the software used for fitting. To validate the software, one often uses pre-generated reference datasets. A reference dataset is associated with a reference solution, which is compared with the solution returned by the software under test, referred to as the test solution.
In real world applications datasets are represented and stored using finite numbers of digits. Using such finite precision, when the software under test processes a dataset, the test solution generally differs from the reference solution. In this paper we present a method for determining the numerical uncertainty of the reference solution that is then used in comparison of the test solution with the reference solution. The results of applying the method to fitting data to a geometric elements are presented.
