Constructing Benchmark Databases and Protocols for Medical Image Analysis: Diabetic Retinopathy

We address the performance evaluation practices for developing medical image analysis methods, in particular, how to establish and share databases of medical images with verified ground truth and solid evaluation protocols. Such databases support the development of better algorithms, execution of profound method comparisons, and, consequently, technology transfer from research laboratories to clinical practice. For this purpose, we propose a framework consisting of reusable methods and tools for the laborious task of constructing a benchmark database. We provide a software tool for medical image annotation helping to collect class label, spatial span, and expert's confidence on lesions and a method to appropriately combine the manual segmentations from multiple experts. The tool and all necessary functionality for method evaluation are provided as public software packages. As a case study, we utilized the framework and tools to establish the DiaRetDB1 V2.1 database for benchmarking diabetic retinopathy detection algorithms. The database contains a set of retinal images, ground truth based on information from multiple experts, and a baseline algorithm for the detection of retinopathy lesions.

[1]  Tomi Kauppi,et al.  Eye Fundus Image Analysis for Automatic Detection of Diabetic Retinopathy , 2010 .

[2]  P. Zimmet,et al.  Definition, diagnosis and classification of diabetes mellitus and its complications. Part 1: diagnosis and classification of diabetes mellitus. Provisional report of a WHO Consultation , 1998, Diabetic medicine : a journal of the British Diabetic Association.

[3]  A Hoover,et al.  Locating blood vessels in retinal images by piece-wise threshold probing of a matched filter response , 1998, AMIA.

[4]  Anil K. Jain,et al.  Unsupervised Learning of Finite Mixture Models , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Joni-Kristian Kämäräinen,et al.  Feature representation and discrimination based on Gaussian mixture model probability densities - Practices and algorithms , 2006, Pattern Recognit..

[6]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[7]  Neil A. Thacker,et al.  Performance characterization in computer vision: A guide to best practices , 2008, Comput. Vis. Image Underst..

[8]  Joni-Kristian Kämäräinen,et al.  Combining Multiple Image Segmentations by Maximizing Expert Agreement , 2012, MLMI.

[9]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[10]  Max A. Viergever,et al.  Ridge-based vessel segmentation in color images of the retina , 2004, IEEE Transactions on Medical Imaging.

[11]  Joni-Kristian Kämäräinen,et al.  A Framework for Constructing Benchmark Databases and Protocols for Retinopathy in Medical Image Analysis , 2012, IScIDE.

[12]  Joni-Kristian Kämäräinen,et al.  Fusion of Multiple Expert Annotations and Overall Score Selection for Medical Image Diagnosis , 2009, SCIA.

[13]  Felipe Orihuela-Espina,et al.  Quantitative analysis of multi-spectral fundus images , 2006, Medical Image Anal..

[14]  Jean-Philippe Thiran,et al.  The BANCA Database and Evaluation Protocol , 2003, AVBPA.

[15]  Hyeonjoon Moon,et al.  The FERET evaluation methodology for face-recognition algorithms , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[16]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  A.D. Hoover,et al.  Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response , 2000, IEEE Transactions on Medical Imaging.

[18]  Lucila Ohno-Machado,et al.  The use of receiver operating characteristic curves in biomedical informatics , 2005, J. Biomed. Informatics.

[19]  Joni-Kristian Kämäräinen,et al.  The DIARETDB1 Diabetic Retinopathy Database and Evaluation Protocol , 2007, BMVC.

[20]  Gunvor von Wendt,et al.  Screening for diabetic retinopathy : Aspects of photographic methods , 2005 .

[21]  Qin Li,et al.  Retinopathy Online Challenge: Automatic Detection of Microaneurysms in Digital Color Fundus Photographs , 2010, IEEE Transactions on Medical Imaging.

[22]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[23]  William M. Wells,et al.  Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation , 2004, IEEE Transactions on Medical Imaging.

[24]  Sushma G. Thorat.  Locating the Optic Nerve in a Retinal Image Using the Fuzzy Convergence of the Blood Vessels , 2014 .

[25]  Maged Habib,et al.  REVIEW - A reference data set for retinal vessel profiles , 2008, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[26]  Jianguo Zhang,et al.  The PASCAL Visual Object Classes Challenge , 2006 .