Accuracy of a Deep Learning System for Classification of Papilledema Severity on Ocular Fundus Photographs

Objective To evaluate the performance of a deep learning system (DLS) in classifying the severity of papilledema associated with increased intracranial pressure on standard retinal fundus photographs. Methods A DLS was trained to automatically classify papilledema severity in 965 patients (2,103 mydriatic fundus photographs), representing a multiethnic cohort of patients with confirmed elevated intracranial pressure. Training was performed on 1,052 photographs with mild/moderate papilledema (MP) and 1,051 photographs with severe papilledema (SP) classified by a panel of experts. The performance of the DLS and that of 3 independent neuro-ophthalmologists were tested in 111 patients (214 photographs, 92 with MP and 122 with SP) by calculating the area under the receiver operating characteristics curve (AUC), accuracy, sensitivity, and specificity. Kappa agreement scores between the DLS and each of the 3 graders and among the 3 graders were calculated. Results The DLS successfully discriminated between photographs of MP and SP, with an AUC of 0.93 (95% confidence interval [CI] 0.89–0.96) and an accuracy, sensitivity, and specificity of 87.9%, 91.8%, and 86.2%, respectively. This performance was comparable with that of the 3 neuro-ophthalmologists (84.1%, 91.8%, and 73.9%, p = 0.19, p = 1, p = 0.09, respectively). Misclassification by the DLS was mainly observed for moderate papilledema (Frisén grade 3). Agreement scores between the DLS and the neuro-ophthalmologists’ evaluation was 0.62 (95% CI 0.57–0.68), whereas the intergrader agreement among the 3 neuro-ophthalmologists was 0.54 (95% CI 0.47–0.62). Conclusions Our DLS accurately classified the severity of papilledema on an independent set of mydriatic fundus photographs, achieving a comparable performance with that of independent neuro-ophthalmologists. Classification of Evidence This study provides Class II evidence that a DLS using mydriatic retinal fundus photographs accurately classified the severity of papilledema associated in patients with a diagnosis of increased intracranial pressure.

[1]  Raymond P. Najjar,et al.  Optic Disc Classification by Deep Learning versus Expert Neuro‐Ophthalmologists , 2020, Annals of neurology.

[2]  J. A. McHugh,et al.  An Update on Imaging in Idiopathic Intracranial Hypertension , 2020, Frontiers in Neurology.

[3]  John J. Chen,et al.  Etiology of Papilledema in Patients in the Eye Clinic Setting , 2020, JAMA network open.

[4]  Raymond P. Najjar,et al.  Artificial Intelligence to Detect Papilledema from Ocular Fundus Photographs. , 2020, The New England journal of medicine.

[5]  N. Newman,et al.  Feasibility of a Nonmydriatic Ocular Fundus Camera in an Outpatient Neurology Clinic , 2020, The neurologist.

[6]  Katy C. Liu,et al.  Presentation and Progression of Papilledema in Cerebral Venous Sinus Thrombosis. , 2020, American journal of ophthalmology.

[7]  Raymond P. Najjar,et al.  Artificial intelligence for detection of optic disc abnormalities. , 2019, Current opinion in neurology.

[8]  R. Jensen,et al.  Predictive role of presenting symptoms and clinical findings in idiopathic intracranial hypertension , 2019, Journal of the Neurological Sciences.

[9]  Chiazor Udochukwu Onyia,et al.  Idiopathic intracranial hypertension: Proposal of a stratification strategy for monitoring risk of disease progression , 2019, Clinical Neurology and Neurosurgery.

[10]  Tuan Leng Tay,et al.  U-Net: deep learning for cell counting, detection, and morphometry , 2018, Nature Methods.

[11]  Beau B. Bruce,et al.  Optic nerve appearance as a predictor of visual outcome in patients with idiopathic intracranial hypertension , 2018, British Journal of Ophthalmology.

[12]  Aaron Y. Lee,et al.  Artificial intelligence and deep learning in ophthalmology , 2018, British Journal of Ophthalmology.

[13]  E. Frew,et al.  The expanding burden of idiopathic intracranial hypertension , 2018, Eye.

[14]  Felipe A. Medeiros,et al.  From Machine to Machine: An OCT-trained Deep Learning Algorithm for Objective Quantification of Glaucomatous Damage in Fundus Photographs , 2018, Ophthalmology.

[15]  M. Abràmoff,et al.  Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices , 2018, npj Digital Medicine.

[16]  Sophie Vanbelle,et al.  Asymptotic variability of (multilevel) multirater kappa coefficients , 2018, Statistical methods in medical research.

[17]  James M. Brown,et al.  Automated Diagnosis of Plus Disease in Retinopathy of Prematurity Using Deep Convolutional Neural Networks , 2018, JAMA ophthalmology.

[18]  L. Vuong,et al.  Ganglion Cell Complex Analysis as a Potential Indicator of Early Neuronal Loss in Idiopathic Intracranial Hypertension , 2018, Neuro-ophthalmology.

[19]  K. Digre,et al.  Idiopathic intracranial hypertension: consensus guidelines on management , 2018, Journal of Neurology, Neurosurgery, and Psychiatry.

[20]  M. Goyal,et al.  Clinical and Neuro-ophthalmologic Predictors of Visual Outcome in Idiopathic Intracranial Hypertension , 2018, Neuro-ophthalmology.

[21]  P. Subramanian,et al.  Headache in Idiopathic Intracranial Hypertension: Findings From the Idiopathic Intracranial Hypertension Treatment Trial , 2017, Headache.

[22]  R. Saladino,et al.  Feasibility of the Digital Retinography System Camera in the Pediatric Emergency Department , 2017, Pediatric emergency care.

[23]  Beau B. Bruce,et al.  Overdiagnosis of idiopathic intracranial hypertension , 2016, Neurology.

[24]  Beau B. Bruce,et al.  Risk factors for poor visual outcome in patients with idiopathic intracranial hypertension , 2015, Neurology.

[25]  M. El-Dairi,et al.  Anatomic and visual function outcomes in paediatric idiopathic intracranial hypertension , 2015, British Journal of Ophthalmology.

[26]  R. Kardon,et al.  Causes and Prognosis of Visual Acuity Loss at the Time of Initial Presentation in Idiopathic Intracranial Hypertension. , 2015, Investigative ophthalmology & visual science.

[27]  S. Feldon,et al.  Photographic Reading Center of the Idiopathic Intracranial Hypertension Treatment Trial (IIHTT): Methods and Baseline Results. , 2015, Investigative ophthalmology & visual science.

[28]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[29]  R. Kardon.  Optical Coherence Tomography in Papilledema: What Am I Missing? , 2014, Journal of neuro-ophthalmology : the official journal of the North American Neuro-Ophthalmology Society.

[30]  M. Moster,et al.  Literature Commentary , 2014 .

[31]  M. Marciani,et al.  Revised diagnostic criteria for the pseudotumor cerebri syndrome in adults and children , 2014, Neurology.

[32]  D. Friedman.  The pseudotumor cerebri syndrome. , 2014, Neurologic clinics.

[33]  David W Wright,et al.  Diagnostic accuracy and use of nonmydriatic ocular fundus photography by emergency physicians: phase II of the FOTO-ED study. , 2013, Annals of emergency medicine.

[34]  M. McHugh.  Interrater reliability: the kappa statistic , 2012, Biochemia medica.

[35]  C. Clarke,et al.  Rating papilloedema: an evaluation of the Frisén classification in idiopathic intracranial hypertension , 2012, Journal of Neurology.

[36]  Gilberto Zamora,et al.  Automated analysis of optic nerve images for detection and staging of papilledema. , 2011, Investigative ophthalmology & visual science.

[37]  R. Rosenman,et al.  Idiopathic intracranial hypertension in the USA: the role of obesity in establishing prevalence and healthcare costs , 2011, Obesity reviews : an official journal of the International Association for the Study of Obesity.

[38]  Beau B. Bruce,et al.  Nonmydriatic ocular fundus photography in the emergency department. , 2011, The New England journal of medicine.

[39]  Michael Wall,et al.  Diagnosis and grading of papilledema in patients with raised intracranial pressure using optical coherence tomography vs clinical expert assessment using a clinical staging scale. , 2010, Archives of ophthalmology.

[40]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  I. Scott,et al.  Expert agreement in evaluating the optic disc for glaucoma. , 1992, Ophthalmology.

[42]  H. S. Thompson,et al.  Visual loss in pseudotumor cerebri. Follow-up of 57 patients from five to 41 years and a profile of 14 patients with permanent severe visual loss. , 1982, Archives of neurology.

[43]  L Frisén,et al.  Swelling of the optic nerve head: a staging scheme , 1982, Journal of neurology, neurosurgery, and psychiatry.

[44]  Geoffrey E. Hinton,et al.  Deep Learning , 2015 .

[45]  M. Sanders,et al.  Factors affecting visual loss in benign intracranial hypertension. , 1984, Ophthalmology.