Enhancement of Retinal Image From Line-Scanning Ophthalmoscope Using Generative Adversarial Networks

A line-scanning ophthalmoscope (LSO) is a retinal imaging technique that has the characteristics of high imaging resolution, wide field of view, and high imaging speed. However, the high-speed imaging with rather short exposure time inevitably reduces the signal intensity, and many factors, such as speckle noise and intraocular scatter, further degrade the signal-to-noise ratio (SNR) of retinal images. To effectively improve the image quality without increasing the LSO system’s complexity, the post-processing method of image super-resolution (SR) is adopted. In this paper, we propose a learning-based multi-frame retinal image SR method that directly learns an end-to-end mapping from low-resolution (LR) image sequences to high-resolution (HR) images. This network was validated on down-sampled and real LSO image sequences. We evaluated the method on a down-sampled dataset with the metrics of peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and perceptual distance. Moreover, the power spectra and full width at half maximum (FWHM) were used as the no-reference image quality assessment (NR-IQA) algorithms to evaluate the reconstruction results of the real LSO image sequences. The experimental results indicate that the proposed method can significantly enhance the SNR of LSO images and efficiently improve the resolution of LSO retinal images, which has great practical significance for clinical diagnosis and analysis.

[1]  Yann LeCun,et al.  Deep multi-scale video prediction beyond mean square error , 2015, ICLR.

[2]  Frédéric Champagnat,et al.  A Fourier interpretation of super-resolution techniques , 2005, IEEE International Conference on Image Processing 2005.

[3]  Jason Cong,et al.  Computed Tomography Image Enhancement using 3D Convolutional Neural Network , 2018, DLMIA/ML-CDS@MICCAI.

[4]  Renjie Liao,et al.  Detail-Revealing Deep Video Super-Resolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[5]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6]  Zengfu Wang,et al.  Video Superresolution via Motion Compensation and Deep Residual Learning , 2017, IEEE Transactions on Computational Imaging.

[7]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[8]  Gilberto Zamora,et al.  Quantitative and qualitative image quality analysis of super resolution images from a low cost scanning laser ophthalmoscope , 2011, Medical Imaging.

[9]  Ming-Chao Chiang,et al.  Super-Resolution via Image Warping , 2002 .

[10]  Deyun Wei,et al.  Image super-resolution reconstruction using the high-order derivative interpolation associated with fractional filter functions , 2016, IET Signal Process..

[11]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[13]  Varsha H. Patil.  Super Resolution in Medical Imaging for An Automated Proficient Process , 2006 .

[14]  Denis Kouame,et al.  Super-resolution in medical imaging : An illustrative approach through ultrasound , 2009, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[15]  Hayit Greenspan,et al.  Super-Resolution in Medical Imaging , 2009, Comput. J..

[16]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[17]  Joachim Hornegger,et al.  Multi-frame Super-resolution with Quality Self-assessment for Retinal Fundus Videos , 2014, MICCAI.

[18]  Jing Wu,et al.  Retinal Fundus Image Enhancement Using the Normalized Convolution and Noise Removing , 2016, Int. J. Biomed. Imaging.

[19]  Khizar Hayat,et al.  Super-Resolution via Deep Learning , 2017, Digit. Signal Process..

[20]  Deyun Wei,et al.  Reconstruction of multidimensional bandlimited signals from multichannel samples in linear canonical transform domain , 2014, IET Signal Process..

[21]  Deyun Wei,et al.  Generalized Sampling Expansions with Multiple Sampling Rates for Lowpass and Bandpass Signals in the Fractional Fourier Transform Domain , 2016, IEEE Transactions on Signal Processing.

[22]  Lixin Zheng,et al.  A Fast Medical Image Super Resolution Method Based on Deep Learning Network , 2019, IEEE Access.

[23]  Dwarikanath Mahapatra,et al.  Image Super Resolution Using Generative Adversarial Networks and Local Saliency Maps for Retinal Image Analysis , 2017, MICCAI.

[24]  Yudong Zhang,et al.  Compact high-speed line scanning quasi-confocal ophthalmoscope and retina imaging experiments , 2014, Other Conferences.

[25]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[26]  Daniel X Hammer,et al.  Line-scanning laser ophthalmoscope. , 2006, Journal of biomedical optics.

[27]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[28]  何益,et al.  Retina imaging by using compact line scanning quasi-confocal ophthalmoscope , 2013 .

[29]  Mei Zhou,et al.  Color Retinal Image Enhancement Based on Luminosity and Contrast Adjustment , 2018, IEEE Transactions on Biomedical Engineering.

[30]  Aggelos K. Katsaggelos,et al.  Generative Adversarial Networks and Perceptual Losses for Video Super-Resolution , 2018, IEEE Transactions on Image Processing.

[31]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  R. Webb,et al.  Confocal scanning laser ophthalmoscope. , 1987, Applied optics.

[33]  Sabine Süsstrunk,et al.  A Frequency Domain Approach to Registration of Aliased Images with Application to Super-resolution , 2006, EURASIP J. Adv. Signal Process..

[34]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[35]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Zhimin Chen,et al.  Face Super-Resolution Through Wasserstein GANs , 2017, ArXiv.

[37]  Daniel L. Rubin,et al.  Differential Data Augmentation Techniques for Medical Imaging Classification Tasks , 2017, AMIA.

[38]  Kaamran Raahemifar,et al.  Comparison of super-resolution algorithms applied to retinal images , 2014, Journal of biomedical optics.

[39]  Aggelos K. Katsaggelos,et al.  Video Super-Resolution With Convolutional Neural Networks , 2016, IEEE Transactions on Computational Imaging.

[40]  Hongming Zhang,et al.  Speckle noise reduction algorithm with total variation regularization in optical coherence tomography. , 2015, Optics express.

[41]  Alan C. Bovik,et al.  A Two-Step Framework for Constructing Blind Image Quality Indices , 2010, IEEE Signal Processing Letters.

[42]  Thomas W. Parks,et al.  Prediction of image detail , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).