Fully Leveraging Deep Learning Methods for Constructing Retinal Fundus Photomontages

Retinal photomontages, which are constructed by aligning and integrating multiple fundus images, are useful in diagnosing retinal diseases affecting peripheral retina. We present a novel framework for constructing retinal photomontages that fully leverage recent deep learning methods. Deep learning based object detection is used to define the order of image registration and blending. Deep learning based vessel segmentation is used to enhance image texture to improve registration performance within a two step image registration framework comprising rigid and non-rigid registration. Experimental evaluation demonstrates the robustness of our montage construction method with an increased amount of successfully integrated images as well as reduction of image artifacts.

[1]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[2]  Philippe C. Cattin,et al.  Objective and expert-independent validation of retinal image registration algorithms by a projective imaging distortion model , 2010, Medical Image Anal..

[3]  Antonis A. Argyros,et al.  An experimental evaluation of the accuracy of keypoints-based retinal image registration , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[4]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[5]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[6]  J. Cunha-Vaz,et al.  Pathophysiology of diabetic retinopathy. , 1978, The British journal of ophthalmology.

[7]  Kyoung Jin Noh,et al.  Multimodal Registration of Fundus Images With Fluorescein Angiography for Fine-Scale Vessel Segmentation , 2020, IEEE Access.

[8]  Philippe C. Cattin,et al.  Retina Mosaicing Using Local Features , 2006, MICCAI.

[9]  Sang Jun Park,et al.  A Novel Fundus Image Reading Tool for Efficient Generation of a Multi-dimensional Categorical Image Database for Machine Learning Algorithm Training , 2018, Journal of Korean medical science.

[10]  Roland E Schmieder,et al.  Wall-to-Lumen Ratio of Retinal Arterioles as a Tool to Assess Vascular Changes , 2009, Hypertension.

[11]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Wei Yang,et al.  Retinal Mosaicking with Vascular Bifurcations Detected on Vessel Mask by a Convolutional Network , 2020, Journal of healthcare engineering.

[13]  Jorge Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[14]  E. Chun,et al.  Predicting High Coronary Artery Calcium Score From Retinal Fundus Images With Deep Learning Algorithms , 2020, Translational vision science & technology.

[15]  B L Trus,et al.  Constructing retinal fundus photomontages. A new computer-based method. , 1996, Investigative ophthalmology & visual science.

[16]  Changming Sun,et al.  Robust Retinal Image Enhancement via Dual-Tree Complex Wavelet Transform and Morphology-Based Method , 2019, IEEE Access.

[17]  D. Roth,et al.  Ranibizumab Induces Regression of Diabetic Retinopathy in Most Patients at High Risk of Progression to Proliferative Diabetic Retinopathy. , 2018, Ophthalmology. Retina.

[18]  Luis Ibáñez,et al.  The Design of SimpleITK , 2013, Front. Neuroinform..

[19]  Sang Jun Park,et al.  Fine-Scale Vessel Extraction in Fundus Images by Registration with Fluorescein Angiography , 2019, MICCAI.

[20]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[21]  Niels D Prins,et al.  Retinal vessel diameters and cerebral small vessel disease: the Rotterdam Scan Study. , 2006, Brain : a journal of neurology.

[22]  Sang Jun Park,et al.  Development and Validation of Deep Learning Models for Screening Multiple Abnormal Findings in Retinal Fundus Images. , 2020, Ophthalmology.

[23]  Andy Lin,et al.  PlantCV v2: Image analysis software for high-throughput plant phenotyping , 2017, PeerJ.

[24]  Sang Jun Park,et al.  Scale-space approximated convolutional neural networks for retinal vessel segmentation , 2019, Comput. Methods Programs Biomed..