EXTERNAL EVALUATION OF ROAD NETWORKS

Internal self-diagnosis and external evaluation of the obtained results are of major importance for the relevance of any automatic system for practical applications. Obviously, this statement is also true for automatic image analysis in photogrammetry and remote sensing. Recently, automatic systems for the extraction of road networks reached a state in which a systematic evaluation of the results seems to be meaningful. This paper deals with the external evaluation of automatic road extraction results by comparison to manually plotted linear road axes used as reference data. The comparison is performed in two steps: (1) Matching of the extracted primitives to the reference network; (2) Calculation of quality measures. Each step depends on the other: the less tolerant is matching, the less exhaustive the extraction is considered to be, but the more accurate it looks. Therefore, matching is an important part of the evaluation process. The quality measures described in this paper comprise measures for the evaluation of the road axes, the network properties, and the crossings. The evaluation methodology is described in detail. Results for the evaluation of simulated as well as real data are presented and discussed. They show the behavior of the quality measures with respect to different deficiencies of the extraction results. Internal self-diagnosis and external evaluation of the obtained results are of major importance for the relevance of automatic systems for practical applications. Obviously, this statement is also true for automatic image analysis in photogrammetry and remote sensing. Recently, automatic systems reached a state in which a systematic evaluation of the results seems to be meaningful. Both, internal self-diagnosis and external evaluation should yield quantitative results which are independent of a human observer. A good description for the result of internal self-diagnosis is the traffic light paradigm (Forstner, 1996): a green light stands for a result found to be correct as far as the diagnosis tool is concerned, a red light means an incorrect result, and a yellow light implies that further probing is necessary. External evaluation needs some kind of reference data and compares them to the automatically obtained results. In this paper we deal with the external evaluation of automatically extracted roads by means of comparison to manually plotted linear road axes used as reference data. Some approaches on the evaluation of image analysis results can be found in the literature. In (McGlone and Shufelt, 1994) and (Hsieh, 1995) the evaluation of automated building extraction is reported. The results of the extraction are pixels (in image space) or voxels (in object space) which are classified as “building” or “non-building”. The degree of overlap between the results of the automated extraction and a manually generated reference is determined by matching of the corresponding pixels or voxels, respectively. Subsequently, measures for quantifying completeness and correctness of the extraction result are calculated. Road data from maps are analyzed with regard to distortions which are induced by the map production process in (Guerin et al., 1995). A data set of the French Topographic Database (BDTopo) is used as reference. The comparison is performed manually. The accuracy of the position of crossroads as well as the orientation of the connected roads, and their number and nature are investigated. Evaluation of the roads concentrates on measures for their geometrical accuracy. In (Airault et al., 1996) an evaluation methodology is proposed which is supposed to quantify the benefits of automatic and semi-automatic road extraction algorithms compared to manual data capture. The measures comprise geometric accuracy, success rate and in particular the time needed for data capture. (Ruskone and Airault, 1997) present the evaluation of a multi-phase automatic road extraction. It points out the benefits of the different phases and quantifies the quality of the overall results. The reference data used is a data set of the BDTopo. Measures are geometric accuracy as well as exhaustivity of the extracted data. In (CMU, 1997, Harvey, 1999) the evaluation is directed towards measuring the quality of (semi-)automatic road extraction with different levels of manual intervention. The reference data is generated by a procedure starting at manually selected positions, followed by automatic road tracking and manual editing. Roads are extracted as regions, and matching of the extracted data with the reference data is carried out using an intersection operation. Only the exhaustivity of the extracted data is further considered. (Fua, 1997) evaluates the effectiveness of different methods for the initialization of ribbon snakes as well as the geometric accuracy of the extracted road data. Manually generated road data serve as reference data. The evaluation focuses on the amount of effort needed by an operator which is measured by the number of necessary mouse actions. Measures for the geometric accuracy of the extracted road data are average and maximum deviation from the reference data. In (Goodchild and Hunter, 1997), the matching of extraction and reference data is carried out using standard GIS functions. From the matching results, measures for the completeness and the correctness of the extraction results are calculated. In Webster’s Dictionary (Webster’s, 1913), a road is defined as follows: