AIROGS: Artificial Intelligence for RObust Glaucoma Screening Challenge

The early detection of glaucoma is essential in preventing visual impairment. Artificial intelligence (AI) can be used to analyze color fundus photographs (CFPs) in a cost-effective manner, making glaucoma screening more accessible. While AI models for glaucoma screening from CFPs have shown promising results in laboratory settings, their performance decreases significantly in real-world scenarios due to the presence of out-of-distribution and low-quality images. To address this issue, we propose the Artificial Intelligence for Robust Glaucoma Screening (AIROGS) challenge. This challenge includes a large dataset of around 113,000 images from about 60,000 patients and 500 different screening centers, and encourages the development of algorithms that are robust to ungradable and unexpected input data. We evaluated solutions from 14 teams in this paper, and found that the best teams performed similarly to a set of 20 expert ophthalmologists and optometrists. The highest-scoring team achieved an area under the receiver operating characteristic curve of 0.99 (95% CI: 0.98-0.99) for detecting ungradable images on-the-fly. Additionally, many of the algorithms showed robust performance when tested on three other publicly available datasets. These results demonstrate the feasibility of robust AI-enabled glaucoma screening.

[1]  S. Li,et al.  A Workflow for Computer-Aided Diagnosis of Glaucoma , 2022, 2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC).

[2]  Jakob Nikolas Kather,et al.  Elevating Fundoscopic Evaluation to Expert Level - Automatic Glaucoma Detection Using Data from the Airogs Challenge , 2022, 2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC).

[3]  H. Bogunović,et al.  Deep Dirichlet Uncertainty for Unsupervised Out-of-Distribution Detection of Eye Fundus Photographs in Glaucoma Screening , 2022, 2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC).

[4]  S. Kasai,et al.  Computer Aided Diagnosis and Out-of-Distribution Detection in Glaucoma Screening Using Color Fundus Photography , 2022, ArXiv.

[5]  Xiaoying Tang,et al.  GAMMA Challenge: Glaucoma grAding from Multi-Modality imAges , 2022, Medical Image Anal..

[6]  Trevor Darrell,et al.  A ConvNet for the 2020s , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ross B. Girshick,et al.  Masked Autoencoders Are Scalable Vision Learners , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  A. Vedaldi,et al.  Open-Set Recognition: A Good Closed-Set Classifier is All You Need , 2021, ICLR.

[9]  Haidar A. Almubarak,et al.  REFUGE2 Challenge: Treasure for Multi-Domain Learning in Glaucoma Assessment , 2022, ArXiv.

[10]  Yixuan Li,et al.  ReAct: Out-of-distribution Detection With Rectified Activations , 2021, NeurIPS.

[11]  Tal Hassner,et al.  Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection , 2021, NeurIPS.

[12]  Quoc V. Le,et al.  EfficientNetV2: Smaller Models and Faster Training , 2021, ICML.

[13]  Lihi Zelnik-Manor,et al.  An Image is Worth 16x16 Words, What is a Video Worth? , 2021, ArXiv.

[14]  Frank Hutter,et al.  TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[15]  Ekin D. Cubuk,et al.  Revisiting ResNets: Improved Training and Scaling Strategies , 2021, NeurIPS.

[16]  Matthieu Cord,et al.  Training data-efficient image transformers & distillation through attention , 2020, ICML.

[17]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[18]  Ariel Kleiner,et al.  Sharpness-Aware Minimization for Efficiently Improving Generalization , 2020, ICLR.

[19]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Yixuan Li,et al.  Energy-based Out-of-distribution Detection , 2020, NeurIPS.

[21]  Lauren Wilcox,et al.  A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy , 2020, CHI.

[22]  Quoc V. Le,et al.  Self-Training With Noisy Student Improves ImageNet Classification , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Xiaoxiao Li,et al.  REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs , 2019, Medical Image Anal..

[24]  Quoc V. Le,et al.  Randaugment: Practical automated data augmentation with a reduced search space , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[25]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  April Y. Maa,et al.  Deep Learning and Glaucoma Specialists: The Relative Importance of Optic Disc Features to Predict Glaucoma Referral in Fundus Photographs. , 2018, Ophthalmology.

[28]  Koenraad A. Vermeer,et al.  Evaluation of an AI system for the automated detection of glaucoma from stereoscopic optic disc photographs: the European Optic Disc Assessment Study , 2019, Eye.

[29]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[30]  Vishal M. Patel,et al.  One-Class Convolutional Neural Network , 2019, IEEE Signal Processing Letters.

[31]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[32]  D. Hood,et al.  Efficacy of a Deep Learning System for Detecting Glaucomatous Optic Neuropathy Based on Color Fundus Photographs. , 2018, Ophthalmology.

[33]  Yuning Jiang,et al.  Unified Perceptual Parsing for Scene Understanding , 2018, ECCV.

[34]  Matthew B. Blaschko,et al.  Towards a glaucoma risk index based on simulated hemodynamics from fundus images , 2018, MICCAI.

[35]  Quoc V. Le,et al.  AutoAugment: Learning Augmentation Policies from Data , 2018, ArXiv.

[36]  M. He,et al.  Efficacy of a Deep Learning System for Detecting Glaucomatous Optic Neuropathy Based on Color Fundus Photographs. , 2018, Ophthalmology.

[37]  E. Finkelstein,et al.  Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With Diabetes , 2017, JAMA.

[38]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[40]  Greg Russell,et al.  DR HAGIS—a fundus image database for the automatic extraction of retinal surface vessels from diabetic patients , 2017, Journal of medical imaging.

[41]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[42]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[44]  A. Tuulonen,et al.  A Systematic Review of End-of-Life Visual Impairment in Open-Angle Glaucoma: An Epidemiological Autopsy , 2016, Journal of glaucoma.

[45]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[47]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[48]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[49]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[50]  Jayanthi Sivaswamy,et al.  A Comprehensive Retinal Image Dataset for the Assessment of Glaucoma from the Optic Nerve Head Analysis , 2015 .

[51]  Tien Yin Wong,et al.  Multiple ocular diseases detection based on joint sparse multi-task learning , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[52]  T. Wong,et al.  Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis. , 2014, Ophthalmology.

[53]  F. Medeiros,et al.  The pathophysiology and treatment of glaucoma: a review. , 2014, JAMA.

[54]  Hidayet Erdöl,et al.  Identification of suitable fundus images using automated quality assessment methods , 2014, Journal of biomedical optics.

[55]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[56]  Paul J. G. Ernest,et al.  Prevalence of end‐of‐life visual impairment in patients followed for glaucoma , 2013, Acta ophthalmologica.

[57]  B. Bengtsson,et al.  Lifetime risk of blindness in open-angle glaucoma. , 2013, American journal of ophthalmology.

[58]  Elli Angelopoulou,et al.  Retinal vessel segmentation by improved matched filtering: evaluation on a new high-resolution fundus image database , 2013, IET Image Process..

[59]  N. Jansonius,et al.  Glaucoma screening during regular optician visits: the feasibility and specificity of screening in real life , 2011, Acta ophthalmologica.

[60]  Francisco Fumero,et al.  RIM-ONE: An open retinal image database for optic nerve evaluation , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[61]  Tien Yin Wong,et al.  ORIGA-light: An online retinal fundus image database for glaucoma analysis and research , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[62]  Jost B Jonas,et al.  Clinical assessment of stereoscopic optic disc photographs for glaucoma: the European Optic Disc Assessment Trial. , 2010, Ophthalmology.

[63]  Jorge A Cuadros,et al.  EyePACS: An Adaptable Telemedicine System for Diabetic Retinopathy Screening , 2009, Journal of diabetes science and technology.

[64]  Arthur P. Dempster,et al.  A Generalization of Bayesian Inference , 1968, Classic Works of the Dempster-Shafer Theory of Belief Functions.

[65]  Miika Linna,et al.  Cost effectiveness and cost utility of an organized screening programme for glaucoma. , 2007, Acta ophthalmologica Scandinavica.

[66]  C. Rutter,et al.  Bootstrap estimation of diagnostic accuracy with patient-clustered data. , 2000, Academic radiology.

[67]  D. McClish.  Analyzing a Portion of the ROC Curve , 1989, Medical decision making : an international journal of the Society for Medical Decision Making.