A Framework for Constructing Benchmark Databases and Protocols for Retinopathy in Medical Image Analysis

We address performance evaluation practices for developing medical image analysis methods, and contribute to the practice to establish and to share databases of medical images with verified ground truth and solid evaluation protocols. This helps to develop better algorithms, to perform profound method comparisons, including the state-of-the-art methods, and consequently, supports technology transfer from research laboratories to clinical practice. For this purpose, we propose a framework consisting of reusable methods and tools for the laborious task of constructing a benchmark database. We provide a medical image annotation software tool which helps to collect and store ground truth for retinopathy lesions from experts, including the fusion of spatial annotations from several experts. The tool and all necessary functionality for method evaluation are provided as a public software package. For demonstration purposes, we utilise the framework and tools to establish the DiaRetDB1 V2.1 database for benchmarking diabetic retinopathy detection algorithms. The database contains a set of retinal images, ground truth from several experts, and a strawman algorithm for the detection of retinopathy lesions.