Training/Test Data Partitioning for Empirical Performance Evaluation

The issue of training/test set design has not gained much attention in works on empirical performance evaluation. Typically, the division of a set of images into training and test set is done manually. But manual partitioning of image sets is always tedious for the human operator and tends to be biased. In this paper it is argued that a systematic, optimization-based approach is advantageous. We formally state the training/test set design task as a set partition problem in an optimization context. Because of the -hardness of this discrete optimization problem, genetic algorithms are proposed to find suboptimal solutions in reasonable time. We use a range image segmentation comparison task as a testbed to do a validation of our approach. The training/test image set design technique proposed in this paper is very general and can be easily applied to other problem domains. Interestingly, it gives us the possibility of generating both “best-case” and “worst-case” test scenarios, thus providing a rich set of tools for evaluating algorithms from various view points.