An Adjustable Error Measure for Image Segmentation Evaluation

Due to the subjective nature of the segmentation process, quantitative evaluation of image segmentation methods is still a difficult task. Humans perceive image objects in different ways. Consequently, human segmentations may come in different levels of refinement, ie, under- and over-segmentations. Popular segmentation error measures in the literature (Arbelaez and OCE) are supervised methods (also called empirical discrepancy methods), in which error is computed by comparing objects in segmentations with a reference (ground-truth) image produced by humans. Since reference images can be many, the key issue for a segmentation error measure is to be consistent in the presence of both under- and over-segmentation. In general, the term consistency refers to the ability of the error measure to be low, when comparing similar segmentations, or high, when faced with different segmentations, while capturing under-or over-segmentations. In this paper we propose a new object-based empirical discrepancy error measure, called Adjustable Object-based Measure (AOM). We introduce a penalty parameter which gives the method the ability to be more (or less) responsive in the presence of over-segmentation. Hence, we extend the notion of consistency so as to include the application's need in the process. Some applications require segmentation to be extremely accurate, hence under-or over-segmentation should be well penalised. Others, do not. By changing the penalty parameter, AOM can deliver more consistent results not only in reference to the under-or over-segmentation issue alone, but also according to the nature of the application. We compare our method with Arbelaez (used as standard measure in the benchmark of Berkeley Segmentation Image Dataset) and OCE. Our results show that AOM not only is more consistent in the presence of over-segmentation, but is also faster to compute. Unlike Arbelaez and OCE, AOM also satisfies the metric axiom of symmetry.

[1]  Hui Zhang,et al.  Image segmentation evaluation: A survey of unsupervised methods , 2008, Comput. Vis. Image Underst..

[2]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[3]  Dmitry B. Goldgof,et al.  Comparison of Edge Detector Performance through Use in an Object Recognition Task , 2001, Comput. Vis. Image Underst..

[4]  Y. J. Zhang,et al.  A survey on evaluation methods for image segmentation , 1996, Pattern Recognit..

[5]  Martial Hebert,et al.  A Measure for Objective Evaluation of Image Segmentation Algorithms , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[6]  Jan J. Gerbrands,et al.  Three-dimensional image segmentation using a split, merge and group approach , 1991, Pattern Recognit. Lett..

[7]  Sang Uk Lee,et al.  On the color image segmentation algorithm based on the thresholding and the fuzzy c-means techniques , 1990, Pattern Recognit..

[8]  Ludmila I. Kuncheva,et al.  An Evaluation Measure of Image Segmentation Based on Object Centres , 2006, ICIAR.

[9]  Jitendra Malik,et al.  From contours to regions: An empirical evaluation , 2009, CVPR.

[10]  Hong Zhang,et al.  An evaluation metric for image segmentation of multiple objects , 2009, Image Vis. Comput..

[11]  Jaime S. Cardoso,et al.  Toward a generic evaluation of image segmentation , 2005, IEEE Transactions on Image Processing.