PERFORMANCE EVALUATION FOR AUTOMATIC FEATURE EXTRACTION

Recent years have seen significant improvements in the performance of automatic cartographic feature extraction (CFE) systems. Early systems, painstakingly tweaked to work in a very limited fashion on small images, have given way to systems which produce credible results in situations representative of production environments. While no existing automatic system is yet ready for incorporation into a production environment, significant strides have been made toward this goal. Semi-automated systems, with significant man-in-the-loop capabilities, continue to be developed by photogrammetric workstation vendors. However, a fundamental requirement for system development, and an absolute prerequisite for production applications, is the rigorous evaluation of automated system performance. Indeed, without meaningful evaluation, we can hardly be said to be doing science. Rigorous evaluation requires the definition of a set of metrics, relevant to user requirements and meaningful in terms of expected system performance. These metrics must be generated across common, well-documented datasets which are representative of production sources and scenes. To provide concrete examples of system evaluation techniques, this paper describes work in the Digital Mapping Laboratory on the evaluation of automated cartographic feature extraction systems. Activities include the definition and publication of metrics for several types of feature extraction systems, the generation and distribution of several large test data sets, and extensive evaluations on our CFE systems. The paper concludes with a discussion of future activities and directions in system evaluation.