REMPE: Registration of Retinal Images Through Eye Modelling and Pose Estimation

Objective: In-vivo assessment of small vessels can promote accurate diagnosis and monitoring of diseases related to vasculopathy, such as hypertension and diabetes. The eye provides a unique, open, and accessible window for directly imaging small vessels in the retina with non-invasive techniques, such as fundoscopy. In this context, accurate registration of retinal images is of paramount importance in the comparison of vessel measurements from original and follow-up examinations, which is required for monitoring the disease and its treatment. At the same time, retinal registration exhibits a range of challenges due to the curved shape of the retina and the modification of imaged tissue across examinations. Thereby, the objective is to improve the state-of-the-art in the accuracy of retinal image registration. Method: In this work, a registration framework that simultaneously estimates eye pose and shape is proposed. Corresponding points in the retinal images are utilized to solve the registration as a 3D pose estimation. Results: The proposed framework is evaluated quantitatively and shown to outperform state-of-the-art methods in retinal image registration for fundoscopy images. Conclusion: Retinal image registration methods based on eye modelling allow to perform more accurate registration than conventional methods. Significance: This is the first method to perform retinal image registration combined with eye modelling. The method improves the state-of-the-art in accuracy of retinal registration for fundoscopy images, quantitatively evaluated in benchmark datasets annotated with ground truth. The implementation of registration method has been made publicly available.

[1]  Gang Wang,et al.  Robust point matching method for multimodal retinal image registration , 2015, Biomed. Signal Process. Control..

[2]  Zahra Hossein-Nejad,et al.  A-RANSAC: Adaptive random sample consensus method in multimodal retinal image registration , 2018, Biomed. Signal Process. Control..

[3]  George K. Matsopoulos,et al.  Multimodal registration of retinal images using self organizing maps , 2004, IEEE Transactions on Medical Imaging.

[4]  Manolis I. A. Lourakis,et al.  Model-Based Pose Estimation for Rigid Objects , 2013, ICVS.

[5]  Afonso Paiva,et al.  Vessel Optimal Transport for Automated Alignment of Retinal Fundus Images , 2019, IEEE Transactions on Image Processing.

[6]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[7]  Jamshid Shanbehzadeh,et al.  An efficient approach for robust multimodal retinal image registration based on UR-SIFT features and PIIFD descriptors , 2013, EURASIP J. Image Video Process..

[8]  Mohd Yamani Idna Idris,et al.  Feature-Based Retinal Image Registration Using D-Saddle Feature , 2017, Journal of healthcare engineering.

[9]  Joseph M. Reinhardt,et al.  Retinal image mosaicing using the radial distortion correction model , 2008, SPIE Medical Imaging.

[10]  Parminder Singh Reel,et al.  Robust retinal image registration using expectation maximisation with mutual information , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[11]  Francesc Moreno-Noguer,et al.  Non-Rigid Graph Registration Using Active Testing Search , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Philippe C. Cattin,et al.  Retina Mosaicing Using Local Features , 2006, MICCAI.

[13]  Konstantina S. Nikita,et al.  Automatic retinal image registration scheme using global optimization techniques , 1999, IEEE Transactions on Information Technology in Biomedicine.

[14]  Antonis A. Argyros,et al.  Retinal image preprocessing, enhancement, and registration , 2019, Computational Retinal Image Analysis.

[15]  Yuanjie Zheng,et al.  Deep Group-Wise Registration for Multi-Spectral Images From Fundus Images , 2019, IEEE Access.

[16]  Chia-Ling Tsai,et al.  Registration of Challenging Image Pairs: Initialization, Estimation, and Decision , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  M. Sonka,et al.  Retinal Imaging and Image Analysis. , 2010, IEEE transactions on medical imaging.

[18]  Jun Huang,et al.  Gaussian field estimator with manifold regularization for retinal image registration , 2019, Signal Process..

[19]  R. Klein,et al.  Methods for evaluation of retinal microvascular abnormalities associated with hypertension/sclerosis in the Atherosclerosis Risk in Communities Study. , 1999, Ophthalmology.

[20]  Daniel Braun,et al.  EyeSLAM: Real‐time simultaneous localization and mapping of retinal vessels during intraocular microsurgery , 2018, The international journal of medical robotics + computer assisted surgery : MRCAS.

[21]  Badrinath Roysam,et al.  Integrated Analysis of Vascular and Nonvascular Changes From Color Retinal Fundus Image Sequences , 2007, IEEE Transactions on Biomedical Engineering.

[22]  Antonis A. Argyros,et al.  Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints , 2011, 2011 International Conference on Computer Vision.

[23]  Jing Tian,et al.  Retinal image registration using topological vascular tree segmentation and bifurcation structures , 2015, Biomed. Signal Process. Control..

[24]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[25]  Koen A. Vermeer,et al.  A Hierarchical Coarse-to-Fine Approach for Fundus Image Registration , 2014, WBIR.

[26]  Erez N. Ribak,et al.  Enhancing retinal images by extracting structural information , 2014 .

[27]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[28]  Antonis A. Argyros,et al.  Retinal image registration based on keypoint correspondences, spherical eye modeling and camera pose estimation , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[29]  Peng Liu,et al.  A Deep Step Pattern Representation for Multimodal Retinal Image Registration , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[30]  G. Timberlake,et al.  Feature-Based Registration of Retinal Images , 1987, IEEE Transactions on Medical Imaging.

[31]  Antonis A. Argyros,et al.  Scalable 3D Tracking of Multiple Interacting Objects , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Fang Zhang,et al.  Automated quality classification of colour fundus images based on a modified residual dense block network , 2019, Signal Image Video Process..

[33]  Chia-Ling Tsai,et al.  The dual-bootstrap iterative closest point algorithm with application to retinal image registration , 2003, IEEE Transactions on Medical Imaging.

[34]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[35]  Eldar abanoviè,et al.  Deep Neural Network-based Feature Descriptor for Retinal Image Registration , 2018, 2018 IEEE 6th Workshop on Advances in Information, Electronic and Electrical Engineering (AIEEE).

[36]  Xenophon Zabulis,et al.  Super resolution for fundoscopy based on 3D image registration , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[37]  Yulong Shen,et al.  Registration and fusion of retinal images-an evaluation study , 2003, IEEE Transactions on Medical Imaging.

[38]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[39]  A L Siam,et al.  Proliferative diabetic retinopathy. , 2020, Bulletin of the Ophthalmological Society of Egypt.

[40]  Jie Tian,et al.  A Partial Intensity Invariant Feature Descriptor for Multimodal Retinal Image Registration , 2010, IEEE Transactions on Biomedical Engineering.

[41]  Nizan Meitav,et al.  Improving retinal image resolution with iterative weighted shift-and-add. , 2011, Journal of the Optical Society of America. A, Optics, image science, and vision.

[42]  V. Harabis,et al.  Hybrid retinal image registration using phase correlation , 2013 .

[43]  Estelle Parra-Denis,et al.  Ophthalmologic Image Registration Based on Shape-Context: Application to Fundus Autofluorescence (FAF) Images , 2008 .

[44]  Antonis A. Argyros,et al.  Retinal image registration through simultaneous camera pose and eye shape estimation , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[45]  Charles V. Stewart,et al.  A Feature-Based Technique for Joint Linear Estimation of High-Order Image-to-Mosaic Transformations: Mosaicing the Curved Human Retina , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Conor Heneghan,et al.  Registration of digital retinal images using landmark correspondence by expectation maximization , 2004, Image Vis. Comput..

[47]  Sajib Saha,et al.  A Two-Step Approach for Longitudinal Registration of Retinal Images , 2016, Journal of Medical Systems.

[48]  T. Wong,et al.  Hypertensive retinopathy revisited: some answers, more questions , 2005, British Journal of Ophthalmology.

[49]  Parvaneh Saeedi,et al.  Robust Weighted Graph Transformation Matching for Rigid and Nonrigid Image Registration , 2012, IEEE Transactions on Image Processing.

[50]  Gérard G. Medioni,et al.  2-D registration and 3-D shape inference of the retinal fundus from fluorescein images , 2008, Medical Image Anal..

[51]  Jiang Liu,et al.  A low-dimensional step pattern analysis algorithm with application to multimodal retinal image registration , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Joachim H. Nagel,et al.  Registration of high-resolution images of the retina , 1992, Medical Imaging.

[53]  Antonis A. Argyros,et al.  Retinal image registration under the assumption of a spherical eye , 2017, Comput. Medical Imaging Graph..

[54]  Gérard G. Medioni,et al.  Retinal image registration from 2D to 3D , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[55]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[56]  Chia-Ling Tsai,et al.  The Edge-Driven Dual-Bootstrap Iterative Closest Point Algorithm for Registration of Multimodal Fluorescein Angiogram Sequence , 2010, IEEE Transactions on Medical Imaging.

[57]  Xing Zhang,et al.  Retinal Fundus Image Registration via Vascular Structure Graph Matching , 2010, Int. J. Biomed. Imaging.

[58]  Antonis A. Argyros,et al.  An experimental evaluation of the accuracy of keypoints-based retinal image registration , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[59]  Stephen J. McKenna,et al.  RERBEE: Robust Efficient Registration via Bifurcations and Elongated Elements Applied to Retinal Fluorescein Angiogram Sequences , 2012, IEEE Transactions on Medical Imaging.

[60]  Jon Atli Benediktsson,et al.  Unsupervised Change-Detection in Retinal Images by a Multiple-Classifier Approach , 2010, MCS.

[61]  Yang Xiang,et al.  Retinal image registration using bifurcation structures , 2011, 2011 18th IEEE International Conference on Image Processing.

[62]  Jun Huang,et al.  Retinal image registration via feature-guided Gaussian mixture model. , 2016, Journal of the Optical Society of America. A, Optics, image science, and vision.

[63]  Antonis A. Argyros,et al.  FIRE: Fundus Image Registration dataset , 2017 .

[64]  Sabalan Daneshvar,et al.  Retinal Image Registration Using Geometrical Features , 2013, Journal of Digital Imaging.

[65]  R. Navarro,et al.  Accommodation-dependent model of the human eye with aspherics. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[66]  Thitiporn Chanwimaluang.  Advanced retinal imaging: Feature extraction, 2-D registration, and 3-D reconstruction , 2006 .

[67]  Manolis I. A. Lourakis,et al.  3D Object Pose Refinement in Range Images , 2015, ICVS.

[68]  Guillaume Noyel,et al.  Registration Of Retinal Images From Public Health By Minimising An Error Between Vessels Using An Affine Model With Radial Distortions , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).