Image segmentation evaluation: a survey of methods

Image segmentation is a prerequisite for image processing. There are many methods for image segmentation, and as a result, a great number of methods for evaluating segmentation results have also been proposed. How to effectively evaluate the quality of image segmentation is very important. In this paper, the existing image segmentation quality evaluation methods are summarized, mainly including unsupervised methods and supervised methods. Based on hot issues, the application of metrics in natural, medical and remote sensing image evaluation is further outlined. In addition, an experimental comparison for some methods were carried out and the effectiveness of these methods was ranked. At the same time, the effectiveness of classical metrics for remote sensing and medical image evaluation is also verified.

[1]  Evgin Göçeri,et al.  Challenges and Recent Solutions for Image Segmentation in the Era of Deep Learning , 2019 .

[2]  Ying Liu,et al.  Unsupervised Segmentation Evaluation Using Area-Weighted Variance and Jeffries-Matusita Distance for Remote Sensing Images , 2018, Remote. Sens..

[3]  Raveendran Paramesran,et al.  Visual Quality Evaluation of Image Object Segmentation: Subjective Assessment and Objective Measure , 2015, IEEE Transactions on Image Processing.

[4]  Theo van Walsum,et al.  An Evaluation of CNN-based Liver Segmentation Methods using Multi-types of CT Abdominal Images from Multiple Medical Centers , 2019, 2019 19th International Symposium on Communications and Information Technologies (ISCIT).

[5]  Touradj Ebrahimi,et al.  MESH: measuring errors between surfaces using the Hausdorff distance , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[6]  Uday Pratap Singh,et al.  Soft computing approaches for image segmentation: a survey , 2018, Multimedia Tools and Applications.

[7]  Evgin Goceri,et al.  Computer-based segmentation, change detection and quantification for lesions in multiple sclerosis , 2017, 2017 International Conference on Computer Science and Engineering (UBMK).

[8]  Olga Kubassova,et al.  General framework for unsupervised evaluation of quality of segmentation results , 2008, 2008 15th IEEE International Conference on Image Processing.

[9]  Hui Zhang,et al.  Image segmentation evaluation: A survey of unsupervised methods , 2008, Comput. Vis. Image Underst..

[10]  Shengwei Zhang,et al.  Local and global evaluation for remote sensing image segmentation , 2017 .

[11]  Peijun Li,et al.  Evaluation of multiscale morphological segmentation of multispectral imagery for land cover classification , 2004, IGARSS 2004. 2004 IEEE International Geoscience and Remote Sensing Symposium.

[12]  Subrata Rakshit,et al.  Statistical evaluation of image segmentation , 2010, 2010 IEEE 2nd International Advance Computing Conference (IACC).

[13]  Zhaoquan Cai,et al.  Unsupervised segmentation evaluation: an edge-based method , 2016, Multimedia Tools and Applications.

[14]  Evgin Goceri,et al.  Vessel segmentation from abdominal magnetic resonance images: adaptive and reconstructive approach , 2016, International journal for numerical methods in biomedical engineering.

[15]  Nilanjan Dey,et al.  Social Group Optimization Supported Segmentation and Evaluation of Skin Melanoma Images , 2018, Symmetry.

[16]  Jan J. Gerbrands,et al.  Three-dimensional image segmentation using a split, merge and group approach , 1991, Pattern Recognit. Lett..

[17]  Yong Dou,et al.  Aircraft Segmentation from Remote Sensing Image by Transferring Natual Image Trained Forground Extraction CNN Model , 2019, 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP).

[18]  Dinggang Shen,et al.  High-Resolution Encoder–Decoder Networks for Low-Contrast Medical Image Segmentation , 2020, IEEE Transactions on Image Processing.

[19]  Amir Alansary,et al.  MRBrainS Challenge: Online Evaluation Framework for Brain Image Segmentation in 3T MRI Scans , 2015, Comput. Intell. Neurosci..

[20]  Lei Zhang,et al.  Evaluation of Segmentation Quality via Adaptive Composition of Reference Segmentations , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Thomas Wittenberg,et al.  Using simulated fluorescence cell micrographs for the evaluation of cell image segmentation algorithms , 2017, BMC Bioinformatics.

[22]  Marjan Abdechiri,et al.  An unsupervised evaluation method based on probability density function , 2010, 2010 IEEE International Symposium on Industrial Electronics.

[23]  Christophe Rosenberger,et al.  Genetic fusion: application to multi-components image segmentation , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[24]  Hélène Laurent,et al.  Unsupervised evaluation of image segmentation application to multi-spectral images , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[25]  D. Ming,et al.  Review on High Spatial Resolution Remote Sensing Image Segmentation Evaluation , 2018, Photogrammetric Engineering & Remote Sensing.

[26]  Nina Sumiko Tomita Hirata,et al.  Image Segmentation Assessment from the Perspective of a Higher Level Task , 2015, 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images.

[27]  José Hiroki Saito,et al.  NSGA2-based method for band selection for supervised segmentation in hyperspectral imaging , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[28]  Hui Zhang,et al.  An entropy-based objective evaluation method for image segmentation , 2003, IS&T/SPIE Electronic Imaging.

[29]  Evgin Goceri,et al.  Automated detection and extraction of skull from MR head images: Preliminary results , 2017, 2017 International Conference on Computer Science and Engineering (UBMK).

[30]  Jesmin F. Khan,et al.  Weighted entropy for segmentation evaluation , 2014 .

[31]  Juan Domingo,et al.  Iteratively Learning a Liver Segmentation Using Probabilistic Atlases: Preliminary Results , 2016, 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA).

[32]  Bo Peng,et al.  A Probabilistic Measure for Quantitative Evaluation of Image Segmentation , 2013, IEEE Signal Processing Letters.

[33]  Jian Yang,et al.  A multi-band approach to unsupervised scale parameter selection for multi-scale image segmentation , 2014 .

[34]  Nikos Komodakis,et al.  Learning to compare image patches via convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Allan Hanbury,et al.  A formal method for selecting evaluation metrics for image segmentation , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[36]  Andrea Vedaldi,et al.  MatConvNet: Convolutional Neural Networks for MATLAB , 2014, ACM Multimedia.

[37]  Evgin Göçeri,et al.  Automatic labeling of portal and hepatic veins from MR images prior to liver transplantation , 2016, Int. J. Comput. Assist. Radiol. Surg..

[38]  Tang Zhaohui,et al.  An unsupervised method for flotation froth image segmentation evaluation base on image gray-level distribution , 2013, Proceedings of the 32nd Chinese Control Conference.

[39]  Jian Yang,et al.  A discrepancy measure for segmentation evaluation from the perspective of object recognition , 2015 .

[40]  Allan Hanbury,et al.  Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool , 2015, BMC Medical Imaging.

[41]  Evgin Göçeri,et al.  Artificial Neural Network Based Abdominal Organ Segmentations: A Review , 2015, 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).

[42]  Ronghua Shang,et al.  Densely Based Multi-Scale and Multi-Modal Fully Convolutional Networks for High-Resolution Remote-Sensing Image Semantic Segmentation , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[43]  Zhenzhong Chen,et al.  Visual Quality Evaluation for Semantic Segmentation: Subjective Assessment Database and Objective Assessment Measure , 2019, IEEE Transactions on Image Processing.

[44]  Pengfei Shan,et al.  Image segmentation method based on K-mean algorithm , 2018, EURASIP Journal on Image and Video Processing.

[45]  Wiro Niessen,et al.  Quantitative evaluation of noise reduction and vesselness filters for liver vessel segmentation on abdominal CTA images , 2015, Physics in medicine and biology.

[46]  Y. J. Zhang,et al.  A survey on evaluation methods for image segmentation , 1996, Pattern Recognit..

[47]  Nikhil R. Pal,et al.  Image thresholding: Some new techniques , 1993, Signal Process..

[48]  Gang Li,et al.  Scale-constrained unsupervised evaluation method for multi-scale image segmentation , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[49]  Barbara Zitová,et al.  Performance evaluation of image segmentation algorithms on microscopic image data , 2015, Journal of microscopy.

[50]  Farid García,et al.  Segmentation of images by color features: A survey , 2018, Neurocomputing.

[51]  Tepán Rubar.  Quality Measurement of Image Segmentation Evaluation Methods , 2012, SITIS.

[52]  Xiaoli Zhang,et al.  A weighted-ROC graph based metric for image segmentation evaluation , 2016, Signal Process..

[53]  Alfredo Illanes,et al.  Evaluation of Commonly Used Algorithms for Thyroid Ultrasound Images Segmentation and Improvement Using Machine Learning Approaches , 2018, Journal of healthcare engineering.

[54]  Bo Peng,et al.  Region Based Exemplar References for Image Segmentation Evaluation , 2016, IEEE Signal Processing Letters.

[55]  Jesmin Khan,et al.  Evaluation of the number of segments using weighted entropy , 2011, 2011 IEEE 43rd Southeastern Symposium on System Theory.

[56]  Oleh Pitsun,et al.  Regions matching algorithms analysis to quantify the image segmentation results , 2016, 2016 XIth International Scientific and Technical Conference Computer Sciences and Information Technologies (CSIT).

[57]  Brian Johnson,et al.  Unsupervised image segmentation evaluation and refinement using a multi-scale approach , 2011 .

[58]  Roberto de Alencar Lotufo,et al.  Benchmark for Quantitative Evaluation of Assisted Object Segmentation Methods to Image Sequences , 2008, 2008 XXI Brazilian Symposium on Computer Graphics and Image Processing.

[59]  Jeffrey J. Rodriguez,et al.  A Ground-Truth Fusion Method for Image Segmentation Evaluation , 2018, 2018 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI).

[60]  Jocelyn Chanussot,et al.  Dynamic Multicontext Segmentation of Remote Sensing Images Based on Convolutional Networks , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[61]  C. Mala,et al.  Analysis and performance evaluation of various image segmentation methods , 2014, 2014 International Conference on Contemporary Computing and Informatics (IC3I).

[62]  Martin D. Levine,et al.  Dynamic Measurement of Computer Generated Image Segmentations , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Evgin Goceri,et al.  Challenges and Recent Solutions for Image Segmentation in the Era of Deep Learning , 2019, 2019 Ninth International Conference on Image Processing Theory, Tools and Applications (IPTA).

[64]  Qingbo Wu,et al.  Segmentation quality evaluation based on multi-scale convolutional neural networks , 2017, 2017 IEEE Visual Communications and Image Processing (VCIP).

[65]  Hossam Faris,et al.  An efficient clustering algorithm based on the k-nearest neighbors with an indexing ratio , 2019, International Journal of Machine Learning and Cybernetics.

[66]  Yongji Wang,et al.  Hybrid Remote Sensing Image Segmentation Considering Intrasegment Homogeneity and Intersegment Heterogeneity , 2020, IEEE Geoscience and Remote Sensing Letters.

[67]  M. Neubert,et al.  Enhanced evaluation of image segmentation results , 2010 .

[68]  R Aarthi,et al.  Segmentation and Evaluation of White Blood Cells using Segmentation Algorithms , 2019, 2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI).

[69]  Shamik Sural,et al.  Evaluation of segmentation techniques using region area and boundary matching information , 2012, J. Vis. Commun. Image Represent..

[70]  David Emms,et al.  Fuzzy Evaluations of Image Segmentations , 2018, IEEE Transactions on Fuzzy Systems.

[71]  P.K Sahoo,et al.  A survey of thresholding techniques , 1988, Comput. Vis. Graph. Image Process..

[72]  Elli Angelopoulou,et al.  Supervised multispectral image segmentation with power watersheds , 2012, 2012 19th IEEE International Conference on Image Processing.

[73]  Martial Hebert,et al.  Toward Objective Evaluation of Image Segmentation Algorithms , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  José Martínez-Aroza,et al.  A measure of quality for evaluating methods of segmentation and edge detection , 2001, Pattern Recognit..

[75]  Xin Yang,et al.  Joint Segment-Level and Pixel-Wise Losses for Deep Learning Based Retinal Vessel Segmentation , 2018, IEEE Transactions on Biomedical Engineering.

[76]  Song Wang,et al.  Image-Segmentation Evaluation From the Perspective of Salient Object Extraction , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[77]  Hui Zhang,et al.  Meta-Evaluation of Image Segmentation Using Machine Learning , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[78]  Horst Bunke,et al.  Distance Measures for Image Segmentation Evaluation , 2006, EURASIP J. Adv. Signal Process..

[79]  Woei-Chyn Chu,et al.  Performance measure characterization for evaluating neuroimage segmentation algorithms , 2009, NeuroImage.

[80]  M. Gurcan,et al.  Automated fluorescent miscroscopic image analysis of PTBP1 expression in glioma , 2017, PloS one.

[81]  Frans Coenen,et al.  Face Occlusion Detection Using Deep Convolutional Neural Networks , 2016, Int. J. Pattern Recognit. Artif. Intell..

[82]  Jaime S. Cardoso,et al.  Toward a generic evaluation of image segmentation , 2005, IEEE Transactions on Image Processing.

[83]  Jayaram K. Udupa,et al.  A critical analysis of the methods of evaluating MRI brain segmentation algorithms , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[84]  Reza Ehsani,et al.  Applications of High-Resolution Imaging for Open Field Container Nursery Counting , 2018, Remote. Sens..

[85]  J. Satheesh Kumar,et al.  Performance Evaluation of Image Segmentation using Objective Methods , 2016 .

[86]  Evgin Goceri,et al.  Diagnosis of Alzheimer's disease with Sobolev gradient‐based optimization and 3D convolutional neural network , 2019, International journal for numerical methods in biomedical engineering.

[87]  Carsten Rother,et al.  Panoptic Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[88]  David S. Doermann,et al.  Unsupervised feature learning framework for no-reference image quality assessment , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[89]  Evgin Goceri,et al.  A comparative performance evaluation of various approaches for liver segmentation from SPIR images , 2015 .

[90]  Chen Wang,et al.  Standardized Evaluation System for Left Ventricular Segmentation Algorithms in 3D Echocardiography , 2016, IEEE Transactions on Medical Imaging.

[91]  Sheng-Jyh Wang,et al.  The use of visible color difference in the quantitative evaluation of color image segmentation , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[92]  Jocelyn Chanussot,et al.  Multiscale stochastic watershed for unsupervised hyperspectral image segmentation , 2009, 2009 IEEE International Geoscience and Remote Sensing Symposium.

[93]  Jordi Pont-Tuset,et al.  Supervised Evaluation of Image Segmentation and Object Proposal Techniques , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[94]  Haitao Zhang,et al.  Color Space Transformation and Multi-Class Weighted Loss for Adhesive White Blood Cell Segmentation , 2020, IEEE Access.

[95]  Clement Atzberger,et al.  On the Objectivity of the Objective Function - Problems with Unsupervised Segmentation Evaluation Based on Global Score and a Possible Remedy , 2017, Remote. Sens..

[96]  Aoxue Li,et al.  Global and Local Saliency Analysis for the Extraction of Residential Areas in High-Spatial-Resolution Remote Sensing Image , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[97]  Xu Liu,et al.  An End-To-End Network for Panoptic Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[98]  Xiaoqian Jiang,et al.  Flexible methods for segmentation evaluation: results from CT-based luggage screening. , 2014, Journal of X-ray science and technology.

[99]  Thierry Cresson,et al.  A multi-criteria evaluation platform for segmentation algorithms , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[100]  Evgin Göçeri,et al.  A comparative evaluation for liver segmentation from spir images and a novel level set method using signed pressure force function , 2013 .

[101]  Yessenia Yari,et al.  Multispectral images segmentation using fuzzy probabilistic local cluster for unsupervised clustering , 2017, 2017 IEEE Latin American Conference on Computational Intelligence (LA-CCI).

[102]  Weiping Ni,et al.  Optimal Segmentation Scale Selection for Object-Based Change Detection in Remote Sensing Images Using Kullback–Leibler Divergence , 2020, IEEE Geoscience and Remote Sensing Letters.

[103]  King Ngi Ngan,et al.  Objectness based unsupervised object segmentation quality evaluation , 2017, 2017 Seventh International Conference on Information Science and Technology (ICIST).

[104]  Vittorio Ferrari,et al.  End-to-End Training of Object Class Detectors for Mean Average Precision , 2016, ACCV.

[105]  Bo Peng,et al.  Region-based image segmentation evaluation via perceptual pooling strategies , 2017, Machine Vision and Applications.

[106]  José-Fernán Martínez,et al.  Fast Evaluation of Segmentation Quality with Parallel Computing , 2017, Sci. Program..

[107]  Evgin Goceri,et al.  Analysis of Deep Networks with Residual Blocks and Different Activation Functions: Classification of Skin Diseases , 2019, 2019 Ninth International Conference on Image Processing Theory, Tools and Applications (IPTA).

[108]  Xueliang Zhang,et al.  Segmentation quality evaluation using region-based precision and recall measures for remote sensing images , 2015 .

[109]  S. Pare,et al.  Image Segmentation Using Multilevel Thresholding: A Research Review , 2019, Iranian Journal of Science and Technology, Transactions of Electrical Engineering.

[110]  King Ngi Ngan,et al.  Jaccard index compensation for object segmentation evaluation , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[111]  Xinran Lv,et al.  Hierarchical Weakly Supervised Learning for Residential Area Semantic Segmentation in Remote Sensing Images , 2020, IEEE Geoscience and Remote Sensing Letters.

[112]  Yee-Hong Yang,et al.  Multiresolution Color Image Segmentation , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[113]  Younes Jabrane,et al.  A novel Gini index based evaluation criterion for image segmentation , 2018, Optik.

[114]  Xuezhi Feng,et al.  An Unsupervised Evaluation Method for Remotely Sensed Imagery Segmentation , 2012, IEEE Geoscience and Remote Sensing Letters.

[115]  Chao Huang,et al.  QualityNet: Segmentation quality evaluation with deep convolutional networks , 2016, 2016 Visual Communications and Image Processing (VCIP).

[116]  Liaoying Zhao,et al.  Different Versions of Entropy Rate Superpixel Segmentation For Hyperspectral Image , 2019, 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP).

[117]  Mohcine Bouksim,et al.  New evaluation method using sampling theory to evaluate 3D segmentation algorithms , 2016, 2016 4th IEEE International Colloquium on Information Science and Technology (CiSt).

[118]  Jayaram K. Udupa,et al.  A General and Balanced Region-Based Metric for Evaluating Medical Image Segmentation Algorithms , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[119]  Stépan Srubar.  Quality Measurement of Image Segmentation Evaluation Methods , 2012, 2012 Eighth International Conference on Signal Image Technology and Internet Based Systems.

[120]  Jacek Jakubowski,et al.  LEFMIS: locally-oriented evaluation framework for medical image segmentation algorithms , 2018, Physics in medicine and biology.

[121]  Jordi Pont-Tuset,et al.  Measures and Meta-Measures for the Supervised Evaluation of Image Segmentation , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[122]  Haibo He,et al.  A visual long-short-term memory based integrated CNN model for fabric defect image classification , 2020, Neurocomputing.

[123]  Hui Li,et al.  A Novel Unsupervised Segmentation Quality Evaluation Method for Remote Sensing Images , 2017, Sensors.

[124]  Arjan Kuijper,et al.  Extended surface distance for local evaluation of 3D medical image segmentations , 2015, The Visual Computer.

[125]  Paola Campadelli,et al.  Quantitative evaluation of color image segmentation results , 1998, Pattern Recognit. Lett..

[126]  Libao Zhang,et al.  Region-of-Interest Extraction Based on Frequency Domain Analysis and Salient Region Detection for Remote Sensing Image , 2014, IEEE Geoscience and Remote Sensing Letters.

[127]  Asheesh Kumar Gautam,et al.  Performance evaluation of Hyperspectral image segmentation implemented by recombination of PCT and Bilateral filter based fused images , 2016, 2016 3rd International Conference on Signal Processing and Integrated Networks (SPIN).

[128]  Daniel Rueckert,et al.  Patch-Based Evaluation of Image Segmentation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[129]  Patrik Kamencay,et al.  Simple comparison of image segmentation algorithms based on evaluation criterion , 2011, Proceedings of 21st International Conference Radioelektronika 2011.

[130]  Sylvie Philipp-Foliguet,et al.  New Criteria for Evaluating Image Segmentation Results , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[131]  Fernando Pereira,et al.  Objective evaluation of video segmentation quality , 2003, IEEE Trans. Image Process..

[132]  Tengfei Su.  AN IMPROVED UNSUPERVISED IMAGE SEGMENTATION EVALUATION APPROACH BASED ON UNDER- AND OVER-SEGMENTATION AWARE , 2018 .

[133]  Evgin Göçeri,et al.  A Neural Network Based Kidney Segmentation from MR Images , 2015, 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).

[134]  David Zhang,et al.  A comparative study on quality assessment of high resolution fingerprint images , 2010, 2010 IEEE International Conference on Image Processing.

[135]  Pramod K. Varshney,et al.  On performance limits of image segmentation algorithms , 2015, Comput. Vis. Image Underst..