Empirical Evaluation Methods in Computer Vision

The following sections are included:

• Foreword

• Contents

We describe a framework for evaluating the performance of range image segmentation algorithms. The framework is intended to be fully automated and to allow objective and relevant comparison of performance. In principle, it could be used to evaluate general image segmentation algorithms, but the framework is demonstrated here using range images. The framework implemented is in a publicly available tar file that includes images, code, and shell scripts. The primary performance metric is the number of regions correctly segmented. The definition of “correctly segmented” is parameterized on the percent of mutual overlap between a segmented region in an image and its corresponding region in a ground truth specification of the image. This work should make it possible to directly compare the performance of range image segmentation algorithms intended either for planar-surface scenes or for curved-surface scenes.

The issue of training/test set design has not gained much attention in works on empirical performance evaluation. Typically, the division of a set of images into training and test set is done manually. But manual partitioning of image sets is always tedious for the human operator and tends to be biased. In this paper it is argued that a systematic, optimization-based approach is advantageous. We formally state the training/test set design task as a set partition problem in an optimization context. Because of the -hardness of this discrete optimization problem, genetic algorithms are proposed to find suboptimal solutions in reasonable time. We use a range image segmentation comparison task as a testbed to do a validation of our approach. The training/test image set design technique proposed in this paper is very general and can be easily applied to other problem domains. Interestingly, it gives us the possibility of generating both “best-case” and “worst-case” test scenarios, thus providing a rich set of tools for evaluating algorithms from various view points.

This study examines the role of Eigenvector selection and Eigenspace distance measures on PCA-based face recognition systems. In particular, it builds on earlier results from the FERET face recognition evaluation studies, which created a large face database (1,196 subjects) and a baseline face recognition system for comparative evaluations. This study looks at using a combinations of traditional distance measures (City-block, Euclidean, Angle, Mahalanobis) in Eigenspace to improve performance in the matching stage of face recognition. A statistically significant improvement is observed for the Mahalanobis distance alone when compared to the other three alone. However, no combinations of these measures appear to perform better than Mahalanobis alone. This study also examines questions of how many Eigenvectors to select and according to what ordering criterion. It compares variations in performance due-to different distance measures and numbers of Eigenvectors. Ordering Eigenvectors according to a like-image difference value rather than their Eigenvalues is also considered.

We describe the development of a vision system to detect natural events in a low-resolution image stream. The work involved the assessment of algorithmic design decisions to maximise detection reliability. This assessment was carried out by comparing measures and estimates made by the system under development with measures obtained independently. We show that even when these independent measures are themselves noisy, their independence can serve to guide rational design decisions and allow performance estimates to be made. Although presented here for one particular system design, we believe that such an approach will be applicable to other situations when an image-based system is to be used in the analysis of natural scenes, in the absence of a precise ground truth.

A novel task-based algorithm performance evaluation technique is proposed for the evaluation of geometry-driven diffusion (GDD) methods used for increasing the signal-to-noise ratio in MR tomograms. It is based on a probabilistic model of stepwise constant image corrupted by uncorrelated Gaussian noise. The maximum likelihood estimates of the distribution parameters of the random variable derived from intensity gradient are used for characterization of staircase image artifacts in diffused images. The proposed evaluation technique incorporates a “gold standard” of the GDD algorithms, defined as a diffusion process governed by ideal values of conductance.

Disparity map estimation is often regarded as one of the most demanding operations in computer vision applications. Several algorithms have been proposed to solve this problem. With such a number of distinct approaches, the question of choosing the most suitable algorithm for a given application is often raised. Few and limited resources can be found in the literature covering this problem. In the following paragraphs it will be presented a comparative analysis of the performance and characteristics of a set of similarity measure algorithms proposed in the literature in the past few years. The obtained results can be regarded as an extremely valuable basis for selecting the most suitable registration algorithm for a given application. The study was focused on the analysis of two distinct aspects: the final matching error and the computational load of each of the considered correlation functions. Besides this comparative study, the advantages of using a pyramidal resolution approach were also considered. This scheme has proved to be effective in reducing the overall computation time and the required number of arithmetic operations, having an insignificant loss in the final matching error.

Modern imaging techniques in medicine have revolutionized the study of anatomy and physiology in man. A central factor in the success and increasingly widespread application of imaging-based approaches in clinical and basic research has been the emergence of sophisticated computational methods for extracting salient information from image data. The utility of image processing has prompted the development of numerous algorithms for medical data, but these have largely remained research tools and few have been incorporated into a clinical workflow. A primary cause of this poor track record is the lack of validation of these methods. A workshop was held at the Image Processing Conference of the SPIE Medical Imaging 2000 Symposium to discuss and stimulate developments in performance characterization research for medical image processing algorithms. This report presents highlights from the workshop presentations and from the panel discussion with the audience.