Retinal Vasculature Segmentation Using Local Saliency Maps and Generative Adversarial Networks For Image Super Resolution

We propose an image super resolution(ISR) method using generative adversarial networks (GANs) that takes a low resolution input fundus image and generates a high resolution super resolved (SR) image upto scaling factor of $16$. This facilitates more accurate automated image analysis, especially for small or blurred landmarks and pathologies. Local saliency maps, which define each pixel's importance, are used to define a novel saliency loss in the GAN cost function. Experimental results show the resulting SR images have perceptual quality very close to the original images and perform better than competing methods that do not weigh pixels according to their importance. When used for retinal vasculature segmentation, our SR images result in accuracy levels close to those obtained when using the original images.

[1]  Amod Jog,et al.  Self Super-Resolution for Magnetic Resonance Images , 2016, MICCAI.

[2]  Dwarikanath Mahapatra,et al.  Registration of dynamic renal MR images using neurobiological model of saliency , 2008, 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[3]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[4]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[5]  Dwarikanath Mahapatra,et al.  A novel hybrid approach for severity assessment of Diabetic Retinopathy in colour fundus images , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[6]  Dwarikanath Mahapatra,et al.  Using Saliency Features for Graphcut Segmentation of Perfusion Kidney Images , 2009 .

[7]  D. Louis Collins,et al.  Non-local MRI upsampling , 2010, Medical Image Anal..

[8]  Dwarikanath Mahapatra,et al.  Combining multiple expert annotations using semi-supervised learning and graph cuts for medical image segmentation , 2016, Comput. Vis. Image Underst..

[9]  M. Abràmoff,et al.  Web-based screening for diabetic retinopathy in a primary care population: the EyeCheck project. , 2005, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[10]  Hayit Greenspan,et al.  Chest pathology detection using deep learning with non-medical training , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[11]  Dwarikanath Mahapatra,et al.  Graph cut based automatic prostate segmentation using learned semantic information , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[12]  Joachim M. Buhmann,et al.  Active learning based segmentation of Crohns disease from abdominal MRI , 2016, Comput. Methods Programs Biomed..

[13]  Daniel Rueckert,et al.  Super-resolution reconstruction of cardiac MRI using coupled dictionary learning , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[14]  Elisa Ricci,et al.  Retinal Blood Vessel Segmentation Using Line Operators and Support Vector Classification , 2007, IEEE Transactions on Medical Imaging.

[15]  Joachim M. Buhmann,et al.  Cardiac LV and RV Segmentation Using Mutual Context Information , 2012, MLMI.

[16]  Dwarikanath Mahapatra,et al.  Joint Registration and Segmentation of Dynamic Cardiac Perfusion Images Using MRFs , 2010, MICCAI.

[17]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[18]  Horst Bischof,et al.  Fast and accurate image upscaling with super-resolution forests , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Dwarikanath Mahapatra,et al.  A CNN based neurobiology inspired approach for retinal image quality assessment , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[20]  Dwarikanath Mahapatra,et al.  Automatic Eye Type Detection in Retinal Fundus Image Using Fusion of Transfer Learning and Anatomical Features , 2016, 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA).

[21]  Chuan Li,et al.  Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Matthew B. Blaschko,et al.  Learning Fully-Connected CRFs for Blood Vessel Segmentation in Retinal Images , 2014, MICCAI.

[24]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[25]  Damon M. Chandler,et al.  ${\bf S}_{3}$: A Spectral and Spatial Measure of Local Perceived Sharpness in Natural Images , 2012, IEEE Transactions on Image Processing.

[26]  Ying Sun,et al.  Rigid Registration of Renal Perfusion Images Using a Neurobiology-Based Visual Saliency Model , 2010, EURASIP J. Image Video Process..

[27]  Abdulmotaleb El-Saddik,et al.  A Real-Time Smart Assistant for Video Surveillance Through Handheld Devices , 2014, ACM Multimedia.

[28]  Dwarikanath Mahapatra.  Groupwise registration of dynamic cardiac perfusion images using temporal dynamics and segmentation information , 2012, Medical Imaging: Image Processing.

[29]  Joachim M. Buhmann,et al.  A Supervised Learning Based Approach to Detect Crohn's Disease in Abdominal MR Volumes , 2012, Abdominal Imaging.

[30]  Joachim M. Buhmann,et al.  Glaucoma detection using entropy sampling and ensemble learning for automatic optic cup and disc segmentation , 2017, Comput. Medical Imaging Graph..

[31]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[32]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Minh N. Do,et al.  Semantic Image Inpainting with Perceptual and Contextual Losses , 2016, ArXiv.

[34]  J. Moran,et al.  Sensation and perception , 1980 .

[35]  Matthieu Cord,et al.  WELDON: Weakly Supervised Learning of Deep Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Dwarikanath Mahapatra,et al.  Semi-supervised Segmentation of Optic Cup in Retinal Fundus Images Using Variational Autoencoder , 2017, MICCAI.

[37]  Xiaoou Tang,et al.  Compression Artifacts Reduction by a Deep Convolutional Network , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[38]  Norberto Malpica,et al.  Single-image super-resolution of brain MR images using overcomplete dictionaries , 2013, Medical Image Anal..

[39]  François Rousseau,et al.  Brain Hallucination , 2008, ECCV.

[40]  Max A. Viergever,et al.  Ridge-based vessel segmentation in color images of the retina , 2004, IEEE Transactions on Medical Imaging.

[41]  André J. W. van der Kouwe,et al.  Example-Based Restoration of High-Resolution Magnetic Resonance Image Acquisitions , 2013, MICCAI.

[42]  Thomas Brox,et al.  Generating Images with Perceptual Similarity Metrics based on Deep Networks , 2016, NIPS.

[43]  Xin Yu,et al.  Ultra-Resolving Face Images by Discriminative Generative Networks , 2016, ECCV.

[44]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[45]  Dwarikanath Mahapatra,et al.  Nonrigid Registration of Dynamic Renal MR Images Using a Saliency Based MRF Model , 2008, MICCAI.

[46]  Dwarikanath Mahapatra,et al.  Coherency Based Spatio-Temporal Saliency Detection for Video Object Segmentation , 2014, IEEE Journal of Selected Topics in Signal Processing.

[47]  Joachim M. Buhmann,et al.  Crohn's disease tissue segmentation from abdominal MRI using semantic information and graph cuts , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[48]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[49]  Vincent Lepetit,et al.  Supervised Feature Learning for Curvilinear Structure Segmentation , 2013, MICCAI.

[50]  Victor S. Lempitsky,et al.  N4-Fields: Neural Network Nearest Neighbor Fields for Image Transforms , 2014, ArXiv.

[51]  M. M. Fraza,et al.  Blood vessel segmentation methodologies in retinal images – A survey , 2015 .

[52]  Joachim M. Buhmann,et al.  Joint segmentation and groupwise registration of cardiac DCE MRI using sparse data representations , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[53]  Joachim M. Buhmann,et al.  Prostate MRI Segmentation Using Learned Semantic Knowledge and Graph Cuts , 2014, IEEE Transactions on Biomedical Engineering.

[54]  Daniel Rueckert,et al.  Cardiac Image Super-Resolution with Global Correspondence Using Multi-Atlas PatchMatch , 2013, MICCAI.

[55]  Joachim M. Buhmann,et al.  A field of experts model for optic cup and disc segmentation from retinal fundus images , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[56]  Joachim M. Buhmann,et al.  Active learning based segmentation of Crohn's disease using principles of visual saliency , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[57]  Joachim M. Buhmann,et al.  Boosting Convolutional Filters with Entropy Sampling for Optic Cup and Disc Image Segmentation from Fundus Images , 2015, MLMI.

[58]  Li Cheng,et al.  Learning to Boost Filamentary Structure Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[59]  A.D. Hoover,et al.  Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response , 2000, IEEE Transactions on Medical Imaging.

[60]  Dwarikanath Mahapatra,et al.  An MRF framework for joint registration and segmentation of natural and perfusion images , 2010, 2010 IEEE International Conference on Image Processing.

[61]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[62]  Joachim M. Buhmann,et al.  Computational modeling for assessment of IBD: To be or not to be? , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[63]  Luc Van Gool,et al.  Deep Retinal Image Understanding , 2016, MICCAI.

[64]  Qin Li,et al.  Retinopathy Online Challenge: Automatic Detection of Microaneurysms in Digital Color Fundus Photographs , 2010, IEEE Transactions on Medical Imaging.

[65]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[67]  Joachim M. Buhmann,et al.  Obtaining Consensus Annotations For Retinal Image Segmentation Using Random Forest And Graph Cuts , 2015 .

[68]  Joachim M. Buhmann,et al.  Automatic Detection and Segmentation of Crohn's Disease Tissues From Abdominal MRI , 2013, IEEE Transactions on Medical Imaging.

[69]  Joachim M. Buhmann,et al.  Automatic cardiac RV segmentation using semantic information with graph cuts , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[70]  Dwarikanath Mahapatra,et al.  Landmark Detection in Cardiac MRI Using Learned Local Image Statistics , 2012, STACOM.

[71]  Joachim M. Buhmann,et al.  Weakly supervised semantic segmentation of Crohn's disease tissues from abdominal MRI , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[72]  Antonio Criminisi,et al.  Image Quality Transfer via Random Forest Regression: Applications in Diffusion MRI , 2014, MICCAI.

[73]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[74]  Dwarikanath Mahapatra,et al.  Retinal Image Quality Classification Using Saliency Maps and CNNs , 2016, MLMI@MICCAI.

[75]  Zhang Li,et al.  Image Registration Based on Autocorrelation of Local Structure , 2016, IEEE Transactions on Medical Imaging.

[76]  Konstantinos Kamnitsas,et al.  Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks , 2016, MICCAI.

[77]  Dwarikanath Mahapatra,et al.  Semi-supervised learning and graph cuts for consensus based medical image segmentation , 2016, Pattern Recognit..

[78]  Dwarikanath Mahapatra,et al.  Cardiac Image Segmentation from Cine Cardiac MRI Using Graph Cuts and Shape Priors , 2013, Journal of Digital Imaging.

[79]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[80]  Joachim M. Buhmann,et al.  Localizing and segmenting Crohn's disease affected regions in abdominal MRI using novel context features , 2013, Medical Imaging.

[81]  Dwarikanath Mahapatra,et al.  MRF-Based Intensity Invariant Elastic Registration of Cardiac Perfusion Images Using Saliency Information , 2011, IEEE Transactions on Biomedical Engineering.

[82]  Stephen Lin,et al.  DeepVessel: Retinal Vessel Segmentation via Deep Learning and Conditional Random Field , 2016, MICCAI.

[83]  Dwarikanath Mahapatra,et al.  Image Quality Classification for DR Screening Using Convolutional Neural Networks , 2016 .

[84]  Joachim M. Buhmann,et al.  Semi-Supervised and Active Learning for Automatic Segmentation of Crohn's Disease , 2013, MICCAI.

[85]  Stefan Winkler,et al.  Motion saliency outweighs other low-level features while watching videos , 2008, Electronic Imaging.

[86]  Jitendra Malik,et al.  Hypercolumns for object segmentation and fine-grained localization , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[87]  Dwarikanath Mahapatra,et al.  Automatic Cardiac Segmentation Using Semantic Information from Random Forests , 2014, Journal of Digital Imaging.

[88]  Dwarikanath Mahapatra,et al.  Skull Stripping of Neonatal Brain MRI: Using Prior Shape Information with Graph Cuts , 2012, Journal of Digital Imaging.

[89]  S. Roy,et al.  Retrieval of MR Kidney Images by Incorporating Shape Information in Histogram of Low Level Features , 2009 .

[90]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[91]  Joachim M. Buhmann,et al.  Crohn's disease segmentation from MRI using learned image priors , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[92]  D. Mahapatra,et al.  Analyzing Training Information From Random Forests for Improved Image Segmentation , 2014, IEEE Transactions on Image Processing.

[93]  Dwarikanath Mahapatra.  Joint Segmentation and Groupwise Registration of Cardiac Perfusion Images Using Temporal Information , 2012, Journal of Digital Imaging.

[94]  Dwarikanath Mahapatra,et al.  Segmentation of Optic Disc and Optic Cup in Retinal Fundus Images Using Coupled Shape Regression , 2016 .

[95]  Dwarikanath Mahapatra,et al.  Image Super Resolution Using Generative Adversarial Networks and Local Saliency Maps for Retinal Image Analysis , 2017, MICCAI.

[96]  Roberto Marcondes Cesar Junior,et al.  Retinal vessel segmentation using the 2-D Gabor wavelet and supervised classification , 2005, IEEE Transactions on Medical Imaging.

[97]  Dwarikanath Mahapatra,et al.  Orientation Histograms as Shape Priors for Left Ventricle Segmentation Using Graph Cuts , 2011, MICCAI.

[98]  Vincent Lepetit,et al.  Projection onto the Manifold of Elongated Structures for Accurate Extraction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[99]  Joan Bruna,et al.  Super-Resolution with Deep Convolutional Sufficient Statistics , 2015, ICLR.

[100]  Dwarikanath Mahapatra,et al.  Integrating Segmentation Information for Improved MRF-Based Elastic Image Registration , 2012, IEEE Transactions on Image Processing.

[101]  Joachim M. Buhmann,et al.  A Supervised Learning Approach for Crohn's Disease Detection Using Higher-Order Image Statistics and a Novel Shape Asymmetry Measure , 2013, Journal of Digital Imaging.

[102]  Seunghoon Hong,et al.  Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[103]  Antonio Criminisi,et al.  Bayesian Image Quality Transfer , 2016, MICCAI.

[104]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[105]  A. Routray,et al.  An Active Snake Model for Classification of Extreme Emotions , 2006, 2006 IEEE International Conference on Industrial Technology.

[106]  Guy Cazuguel,et al.  TeleOphta: Machine learning and image processing methods for teleophthalmology , 2013 .

[107]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[108]  Joachim M. Buhmann,et al.  Visual Saliency Based Active Learning for Prostate MRI Segmentation , 2015, MLMI.

[109]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[110]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[111]  Dwarikanath Mahapatra,et al.  Cardiac MRI Segmentation Using Mutual Context Information from Left and Right Ventricle , 2013, Journal of Digital Imaging.

[112]  Dwarikanath Mahapatra,et al.  Segmentation of optic disc and optic cup in retinal fundus images using shape regression , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[113]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[114]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[115]  Dwarikanath Mahapatra,et al.  Illumination invariant tracking in office environments using neurobiology-saliency based particle filter , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[116]  Bunyarit Uyyanonvara,et al.  An Ensemble Classification-Based Approach Applied to Retinal Blood Vessel Segmentation , 2012, IEEE Transactions on Biomedical Engineering.

[117]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[118]  Yael Pritch,et al.  Saliency filters: Contrast based filtering for salient region detection , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.