Artificial Intelligence for Automated Overlay of Fundus Camera and Scanning Laser Ophthalmoscope Images

Purpose The purpose of this study was to evaluate the ability to align two types of retinal images taken on different platforms; color fundus (CF) photographs and infrared scanning laser ophthalmoscope (IR SLO) images using mathematical warping and artificial intelligence (AI). Methods We collected 109 matched pairs of CF and IR SLO images. An AI algorithm utilizing two separate networks was developed. A style transfer network (STN) was used to segment vessel structures. A registration network was used to align the segmented images to each. Neither network used a ground truth dataset. A conventional image warping algorithm was used as a control. Software displayed image pairs as a 5 × 5 checkerboard grid composed of alternating subimages. This technique permitted vessel alignment determination by human observers and 5 masked graders evaluated alignment by the AI and conventional warping in 25 fields for each image. Results Our new AI method was superior to conventional warping at generating vessel alignment as judged by masked human graders (P < 0.0001). The average number of good/excellent matches increased from 90.5% to 94.4% with AI method. Conclusions AI permitted a more accurate overlay of CF and IR SLO images than conventional mathematical warping. This is a first step toward developing an AI that could allow overlay of all types of fundus images by utilizing vascular landmarks. Translational Relevance The ability to align and overlay imaging data from multiple instruments and manufacturers will permit better analysis of this complex data helping understand disease and predict treatment.

[1]  Deok Won Kim,et al.  The possibility of the combination of OCT and fundus images for improving the diagnostic accuracy of deep learning for age-related macular degeneration: a preliminary experiment , 2018, Medical & Biological Engineering & Computing.

[2]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[3]  H. Heimann,et al.  Near-infrared reflectance and autofluorescence imaging characteristics of choroidal nevi , 2016, Eye.

[4]  Rui Bernardes,et al.  Multimodal macula mapping: a new approach to study diseases of the macula. , 2002, Survey of ophthalmology.

[5]  Daniel S. Kermany,et al.  Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning , 2018, Cell.

[6]  Richard B. Rosen,et al.  Combined Three-Dimensional Spectral OCT/SLO Topography and Microperimetry: Steps toward Achieving Functional Spectral OCT/SLO , 2009, Ophthalmic Research.

[7]  Francesco Bandello,et al.  Recent advances in the management of dry age-related macular degeneration: A review , 2017, F1000Research.

[8]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[9]  Geraint Rees,et al.  Clinically applicable deep learning for diagnosis and referral in retinal disease , 2018, Nature Medicine.

[10]  William R. Freeman,et al.  Comparison of conventional color fundus photography and multicolor imaging in choroidal or retinal lesions , 2018, Graefe's Archive for Clinical and Experimental Ophthalmology.

[11]  Dwarikanath Mahapatra,et al.  Deformable medical image registration using generative adversarial networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[12]  C. N. Vidinova,et al.  Fundusautofluoreszenz bei der trockenen Form der altersbedingten Makuladegeneration – Bedeutung für die Prognose , 2013, Klin Monatsbl Augenheilkd.

[13]  Jun Huang,et al.  Retinal image registration via feature-guided Gaussian mixture model. , 2016, Journal of the Optical Society of America. A, Optics, image science, and vision.

[14]  Hang,et al.  Multi-modal and multi-vendor retina image registration , 2018 .

[15]  Jui-Kai Wang,et al.  Multimodal Segmentation of Optic Disc and Cup From SD-OCT and Color Fundus Photographs Using a Machine-Learning Graph-Based Approach , 2015, IEEE Transactions on Medical Imaging.

[16]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[17]  Glenn J Jaffe,et al.  Baseline predictors for one-year visual outcomes with ranibizumab or bevacizumab for neovascular age-related macular degeneration. , 2013, Ophthalmology.

[18]  Martin Hammer,et al.  Fluorescence Lifetime Imaging Ophthalmoscopy (FLIO) of Macular Pigment , 2018, Investigative ophthalmology & visual science.

[19]  Manuel G. Penedo,et al.  Retinal Verification Using a Feature Points-Based Biometric Pattern , 2009, EURASIP J. Adv. Signal Process..

[20]  F. Arcadu,et al.  Deep Learning Predicts OCT Measures of Diabetic Macular Thickening From Color Fundus Photographs. , 2019, Investigative ophthalmology & visual science.

[21]  Jirí Jan,et al.  Registration and Fusion of the Autofluorescent and Infrared Retinal Images , 2008, Int. J. Biomed. Imaging.

[22]  Truong Q. Nguyen,et al.  Joint Vessel Segmentation and Deformable Registration on Multi-Modal Retinal Images Based on Style Transfer , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[23]  Marcos Ortega,et al.  Multimodal Registration of Retinal Images Using Domain-Specific Landmarks and Vessel Enhancement , 2018, KES.

[24]  Volker Tresp,et al.  Predicting Visual Acuity by Using Machine Learning in Patients Treated for Neovascular Age-Related Macular Degeneration. , 2018, Ophthalmology.

[25]  Truong Q. Nguyen,et al.  A Segmentation Based Robust Deep Learning Framework for Multimodal Retinal Image Registration , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[26]  Vincent Lepetit,et al.  Learning to Find Good Correspondences , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Paul Mitchell,et al.  The Progression of Geographic Atrophy Secondary to Age-Related Macular Degeneration. , 2017, Ophthalmology.

[28]  Tomasz Malisiewicz,et al.  SuperPoint: Self-Supervised Interest Point Detection and Description , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Michael D Abràmoff,et al.  Multimodal registration of SD-OCT volumes and fundus photographs using histograms of oriented gradients. , 2016, Biomedical optics express.

[30]  Donald C Hood,et al.  Near-infrared autofluorescence: its relationship to short-wavelength autofluorescence and optical coherence tomography in recessive stargardt disease. , 2015, Investigative ophthalmology & visual science.

[31]  Max A. Viergever,et al.  Ridge-based vessel segmentation in color images of the retina , 2004, IEEE Transactions on Medical Imaging.

[32]  Michael Brady,et al.  MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration , 2012, Medical Image Anal..

[33]  Michael Kalloniatis,et al.  Infrared reflectance imaging in age‐related macular degeneration , 2016, Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians.

[34]  Steffen Schmitz-Valckenberg,et al.  Fundus autofluorescence imaging in dry AMD: 2014 Jules Gonin lecture of the Retina Research Foundation , 2014, Graefe's Archive for Clinical and Experimental Ophthalmology.