A framework for evaluating image segmentation algorithms

The purpose of this paper is to describe a framework for evaluating image segmentation algorithms. Image segmentation consists of object recognition and delineation. For evaluating segmentation methods, three factors-precision (reliability), accuracy (validity), and efficiency (viability)-need to be considered for both recognition and delineation. To assess precision, we need to choose a figure of merit, repeat segmentation considering all sources of variation, and determine variations in figure of merit via statistical analysis. It is impossible usually to establish true segmentation. Hence, to assess accuracy, we need to choose a surrogate of true segmentation and proceed as for precision. In determining accuracy, it may be important to consider different 'landmark' areas of the structure to be segmented depending on the application. To assess efficiency, both the computational and the user time required for algorithm training and for algorithm execution should be measured and analyzed. Precision, accuracy, and efficiency factors have an influence on one another. It is difficult to improve one factor without affecting others. Segmentation methods must be compared based on all three factors, as illustrated in an example wherein two methods are compared in a particular application domain. The weight given to each factor depends on application.

[1]  Jayaram K. Udupa,et al.  Interactive segmentation and boundary surface formation for 3-D digital images , 1982, Comput. Graph. Image Process..

[2]  C E Metz,et al.  Evaluation of receiver operating characteristic curve data in terms of information theory, with applications in radiography. , 1973, Radiology.

[3]  M. Horsfield,et al.  Intra- and inter-observer agreement of brain MRI lesion volume measurements in multiple sclerosis. A comparison of techniques. , 1995, Brain : a journal of neurology.

[4]  Daniel P. Huttenlocher,et al.  Comparing Images Using the Hausdorff Distance , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Max A. Viergever,et al.  A methodology for the validation of image segmentation methods , 1992, [1992] Proceedings Fifth Annual IEEE Symposium on Computer-Based Medical Systems.

[6]  Jerry L Prince,et al.  Current methods in medical image segmentation. , 2000, Annual review of biomedical engineering.

[7]  John D. Austin,et al.  Adaptive histogram equalization and its variations , 1987 .

[8]  Stephen M. Pizer,et al.  Intensity mappings to linearize display devices , 1981 .

[9]  Guido Gerig,et al.  Nonlinear anisotropic filtering of MRI data , 1992, IEEE Trans. Medical Imaging.

[10]  Benoit M. Dawant,et al.  Automatic 3D segmentation of internal structures of the head in MR images using a combination of similarity and free-form transformations , 1998, Medical Imaging.

[11]  Andrew W. Fitzgibbon,et al.  An Experimental Comparison of Range Image Segmentation Algorithms , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Jayaram K. Udupa,et al.  Automatic bone-free rendering of cerebral aneurysms via 3D CTA , 2001, SPIE Medical Imaging.

[13]  Y. J. Zhang,et al.  A survey on evaluation methods for image segmentation , 1996, Pattern Recognit..

[14]  C Busch,et al.  Three-dimensional modeling of biopsy protocols for localized prostate cancer. , 1998, Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society.

[15]  Sankar K. Pal,et al.  A review on image segmentation techniques , 1993, Pattern Recognit..

[16]  J. B. Kneeland,et al.  Analysis of in vivo 3-D internal kinematics of the joints of the foot [MRI analysis] , 1998, IEEE Transactions on Biomedical Engineering.

[17]  Jayaram K. Udupa,et al.  New automatic mode of visualizing the colon via Cine CT , 2001, SPIE Medical Imaging.

[18]  D. Paty,et al.  Interferon beta‐1b is effective in relapsing‐remitting multiple sclerosis , 1993, Neurology.

[19]  Jayaram K. Udupa,et al.  Protocol-independent brain MRI segmentation method , 2002, SPIE Medical Imaging.

[20]  A. C. Rencher.  Methods of multivariate analysis , 1995 .

[21]  Jayaram K. Udupa,et al.  Scale-Based Fuzzy Connected Image Segmentation: Theory, Algorithms, and Validation , 2000, Comput. Vis. Image Underst..

[22]  P. Duquette,et al.  Interferon beta-1b is effective in relapsing-remitting multiple sclerosis. I. Clinical results of a multicenter, randomized, double-blind, placebo-controlled trial. The IFNB Multiple Sclerosis Study Group. , 1993 .

[23]  Alejandro F. Frangi,et al.  Active shape model segmentation with optimal features , 2002, IEEE Transactions on Medical Imaging.

[24]  Jayaram K. Udupa,et al.  An in vivo analysis of the motion of the peri-talar joint complex based on MR imaging , 2001, IEEE Transactions on Biomedical Engineering.

[25]  William A. Yasnoff,et al.  Error measures for scene segmentation , 1977, Pattern Recognit..

[26]  Supun Samarasekera,et al.  Fuzzy Connectedness and Object Definition: Theory, Algorithms, and Applications in Image Segmentation , 1996, CVGIP Graph. Model. Image Process..

[27]  Jayaram K. Udupa,et al.  User-Steered Image Segmentation Paradigms: Live Wire and Live Lane , 1998, Graph. Model. Image Process..

[28]  E G McFarland,et al.  Spiral CT colonography: reader agreement and diagnostic performance with two- and three-dimensional image-display techniques. , 2001, Radiology.

[29]  Jayaram K. Udupa,et al.  Relative Fuzzy Connectedness among Multiple Objects: Theory, Algorithms, and Applications in Image Segmentation , 2001, Comput. Vis. Image Underst..

[30]  Supun Samarasekera,et al.  Multiple sclerosis lesion quantification using fuzzy-connectedness principles , 1997, IEEE Transactions on Medical Imaging.

[31]  M S Brown,et al.  Knowledge-Based Segmentation of Pediatric Kidneys in CT for Measurement of Parenchymal Volume , 2001, Journal of computer assisted tomography.

[32]  Jayaram K. Udupa,et al.  Methodology for evaluating image-segmentation algorithms , 2002, SPIE Medical Imaging.

[33]  X FalcãoAlexandre,et al.  User-steered image segmentation paradigms , 1998 .

[34]  Kevin W. Bowyer,et al.  Validation of Medical Image Analysis Techniques , 2000 .

[35]  Jayaram K. Udupa,et al.  Artery-vein separation via MRA-An image processing approach , 2001, IEEE Transactions on Medical Imaging.

[36]  C. Pelizzari,et al.  Accurate Three‐Dimensional Registration of CT, PET, and/or MR Images of the Brain , 1989, Journal of computer assisted tomography.

[37]  Sargur N. Srihari,et al.  Boundary Detection in Multidimensions , 1982, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  William M. Wells,et al.  Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation , 2004, IEEE Transactions on Medical Imaging.

[39]  Stephen M. Pizer,et al.  Robust object representation through object-relevant use of scale , 1994, Medical Imaging.

[40]  L G Nyúl,et al.  On standardizing the MR image intensity scale , 1999, Magnetic resonance in medicine.

[41]  Jayaram K. Udupa,et al.  A 3D generalization of user-steered live-wire segmentation , 2000, Medical Image Anal..

[42]  D. Louis Collins,et al.  Design and construction of a realistic digital brain phantom , 1998, IEEE Transactions on Medical Imaging.

[43]  D. Miller,et al.  Volume measurement of multiple sclerosis lesions with magnetic resonance images , 1992, Neuroradiology.

[44]  L. Bhar,et al.  MULTIVARIATE ANALYSIS OF VARIANCE , 2003 .

[45]  Aaron Fenster,et al.  Technique for evaluation of semiautomatic segmentation methods , 1999, Medical Imaging.

[46]  Yongmin Kim,et al.  A methodology for evaluation of boundary detection algorithms on medical images , 1997, IEEE Transactions on Medical Imaging.