On the use of the overlapping area matrix for image segmentation evaluation: A survey and new performance measures

The development of common and reasonable criteria for evaluating and comparing the performance of segmentation algorithms has always been a concern for researchers in the area. As it is discussed in the paper, some of the measures proposed are not adequate for general images (i.e. images of any sort of scene, without any assumption about the features of the scene objects or the illumination distribution) because they assume a certain distribution of pixel gray-level or colour values for the interior of the regions. This paper reviews performance measures not performing such an assumption and proposes a set of new performance measures in the same line, called the percentage of correctly grouped pixels (CG), the percentage of over-segmentation (OS) and the percentage of under-segmentation (US). Apart from accounting for misclassified pixels, the proposed set of new measures are intended to compute the level of fragmentation of reference regions into output regions and vice versa. A comparison involving similar measures is provided at the end of the paper.

[1]  William A. Yasnoff,et al.  Error measures for scene segmentation , 1977, Pattern Recognit..

[2]  Hugues Benoit-Cattin,et al.  New discrepancy measures for segmentation evaluation , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[3]  José Martínez-Aroza,et al.  A measure of quality for evaluating methods of segmentation and edge detection , 2001, Pattern Recognit..

[4]  Andrew W. Fitzgibbon,et al.  An Experimental Comparison of Range Image Segmentation Algorithms , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Yee-Hong Yang,et al.  Multiresolution Color Image Segmentation , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Yitzhak Yitzhaky,et al.  A Method for Objective Edge Detection Evaluation and Detector Parameter Selection , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  F. Heyden.  Evaluation of edge detection algorithms , 1989 .

[8]  Nikhil R. Pal,et al.  Image thresholding: Some new techniques , 1993, Signal Process..

[9]  Qian Huang,et al.  Quantitative methods of evaluating image segmentation , 1995, Proceedings., International Conference on Image Processing.

[10]  Robert M. Haralick,et al.  Performance characterization of edge detectors , 1992, Defense, Security, and Sensing.

[11]  Yu-Jin Zhang,et al.  Segmentation evaluation and comparison: a study of various algorithms , 1993, Other Conferences.

[12]  Sean Dougherty,et al.  Edge Detector Evaluation Using Empirical ROC Curves , 2001, Comput. Vis. Image Underst..

[13]  P.K Sahoo,et al.  A survey of thresholding techniques , 1988, Comput. Vis. Graph. Image Process..

[14]  A. Baddeley.  An Error Metric for Binary Images , 1992 .

[15]  Linda G. Shapiro,et al.  Image Segmentation Techniques , 1984, Other Conferences.

[16]  Hugues Benoit-Cattin,et al.  Scalable discrepancy measures for segmentation evaluation , 2002, Proceedings. International Conference on Image Processing.

[17]  Martin D. Levine,et al.  Dynamic Measurement of Computer Generated Image Segmentations , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Azriel Rosenfeld,et al.  Threshold Evaluation Techniques , 1978, IEEE Transactions on Systems, Man, and Cybernetics.

[19]  Jan J. Gerbrands,et al.  Segmentation evaluation using ultimate measurement accuracy , 1992, Electronic Imaging.

[20]  Bruce A. Draper,et al.  Inconsistencies in edge detector evaluation , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[21]  Paola Campadelli,et al.  Quantitative evaluation of color image segmentation results , 1998, Pattern Recognit. Lett..

[22]  Yu Jin Zhang,et al.  Influence of segmentation over feature measurement , 1995, Pattern Recognit. Lett..

[23]  Alberto Ortiz Rodríguez.  New segmentation and edge detection methods using physics-based models of image formation , 2005 .

[24]  Dmitry B. Goldgof,et al.  Comparison of Edge Detector Performance through Use in an Object Recognition Task , 2001, Comput. Vis. Image Underst..

[25]  Theo Gevers,et al.  Classifying color edges in video into shadow-geometry, highlight, or material transitions , 2003, IEEE Trans. Multim..

[26]  M. Strintzis,et al.  Still Image Objective Segmentation Evaluation using Ground Truth , 2003 .

[27]  Sang Uk Lee,et al.  On the color image segmentation algorithm based on the thresholding and the fuzzy c-means techniques , 1990, Pattern Recognit..

[28]  W. Andy Wright,et al.  ROC Method for the Evaluation of Multi-class Segmentation/Classification Algorithms with Infrared Imagery , 2002, BMVC.

[29]  Bernard F. Buxton,et al.  Modelling of single mode distributions of colour data using directional statistics , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[30]  Yasnoff Wa,et al.  Scene-segmentation algorithm development using error measures. , 1984 .

[31]  Theo Gevers,et al.  Robust segmentation and tracking of colored objects in video , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[32]  Timothy F. Cootes,et al.  Improving identification performance by integrating evidence from sequences , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[33]  K. P. B. Thomson,et al.  The evaluation of segmentation results and the overlapping area matrix , 1997 .

[34]  Rita Cucchiara,et al.  Tuning Range Image Segmentation by Genetic Algorithm , 2003, EURASIP J. Adv. Signal Process..

[35]  Jan J. Gerbrands,et al.  Three-dimensional image segmentation using a split, merge and group approach , 1991, Pattern Recognit. Lett..

[36]  Theo Gevers,et al.  Robust Photometric Invariant Region Detection in Multispectral Images , 2003, International Journal of Computer Vision.

[37]  W. Yasnoff,et al.  Scene-segmentation algorithm development using error measures. , 1984, Analytical and quantitative cytology.

[38]  Hugh G. Lewis,et al.  A generalized confusion matrix for assessing area estimates from remotely sensed data , 2001 .

[39]  Y. J. Zhang,et al.  A survey on evaluation methods for image segmentation , 1996, Pattern Recognit..

[40]  Stephen V. Stehman,et al.  Selecting and interpreting measures of thematic classification accuracy , 1997 .

[41]  Jan J. Gerbrands,et al.  Objective and quantitative segmentation evaluation and comparison , 1994, Signal Process..

[42]  John A. Marchant,et al.  On the Performance Characterisation of Image Segmenation Algorithms: A Case Study , 2000, ECCV.

[43]  Theo Gevers,et al.  Adaptive Image Segmentation by Combining Photometric Invariant Region and Edge Information , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  Francisco José Madrid-Cuevas,et al.  Characterization of empirical discrepancy evaluation measures , 2004, Pattern Recognit. Lett..