A Review of the Deep Learning Methods for Medical Images Super Resolution Problems

Abstract Super resolution problems are widely discussed in medical imaging. Spatial resolution of medical images are not sufficient due to the constraints such as image acquisition time, low irradiation dose or hardware limits. To address these problems, different super resolution methods have been proposed, such as optimization or learning-based approaches. Recently, deep learning methods become a thriving technology and are developing at an exponential speed. We think it is necessary to write a review to present the current situation of deep learning in medical imaging super resolution. In this paper, we first briefly introduce deep learning methods, then present a number of important deep learning approaches to solve super resolution problems, different architectures as well as up-sampling operations will be introduced. Afterwards, we focus on the applications of deep learning methods in medical imaging super resolution problems, the challenges to overcome will be presented as well.

[1]  Konstantinos Kamnitsas,et al.  Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks , 2016, MICCAI.

[2]  Peter A. Calabresi,et al.  A Deep Learning Based Anti-aliasing Self Super-Resolution Algorithm for MRI , 2018, MICCAI.

[3]  Jaakko Lehtinen,et al.  Noise2Noise: Learning Image Restoration without Clean Data , 2018, ICML.

[4]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Babajide O Ayinde,et al.  Regularizing Deep Neural Networks by Enhancing Diversity in Feature Extraction , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[6]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[7]  Yu Qiao,et al.  Suppressing Model Overfitting for Image Super-Resolution Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  Jakub Nalepa,et al.  Deep Learning for Multiple-Image Super-Resolution , 2019, IEEE Geoscience and Remote Sensing Letters.

[9]  YiNan Zhang,et al.  Deep Learning- and Transfer Learning-Based Super Resolution Reconstruction from Single Medical Image , 2017, Journal of healthcare engineering.

[10]  Seungyong Lee,et al.  SRFeat: Single Image Super-Resolution with Feature Discrimination , 2018, ECCV.

[11]  Junko Ota,et al.  Application of Super-Resolution Convolutional Neural Network for Enhancing Image Resolution in Chest CT , 2018, Journal of Digital Imaging.

[12]  Jeffrey L. Gunter,et al.  Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks , 2018, SASHIMI@MICCAI.

[13]  Zhongyuan Wang,et al.  Video Satellite Imagery Super Resolution via Convolutional Neural Networks , 2017, IEEE Geoscience and Remote Sensing Letters.

[14]  Quoc V. Le,et al.  AutoAugment: Learning Augmentation Policies from Data , 2018, ArXiv.

[15]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[16]  Taiji Suzuki,et al.  Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks , 2018, 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA).

[17]  Christian Ledig,et al.  Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize , 2017, ArXiv.

[18]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Peter Corcoran,et al.  Smart Augmentation Learning an Optimal Data Augmentation Strategy , 2017, IEEE Access.

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Debiao Li,et al.  Efficient and Accurate MRI Super-Resolution using a Generative Adversarial Network and 3D Multi-Level Densely Connected Network , 2018, MICCAI.

[22]  Jacek M. Zurada,et al.  Deep Learning of Constrained Autoencoders for Enhanced Understanding of Data , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[24]  Jan Kautz,et al.  Loss Functions for Neural Networks for Image Processing , 2015, ArXiv.

[25]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[26]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[27]  Chi-Hieu Pham,et al.  Brain MRI super-resolution using deep 3D convolutional networks , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[28]  Shihui Ying,et al.  Super-resolution reconstruction of MR image with a novel residual learning network algorithm , 2018, Physics in medicine and biology.

[29]  Aaron Carass,et al.  Self super-resolution for magnetic resonance images using deep networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[30]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[31]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[32]  Nanning Zheng,et al.  Incorporating image priors with deep convolutional neural networks for image super-resolution , 2016, Neurocomputing.

[33]  Michael Unser,et al.  Deep Convolutional Neural Network for Inverse Problems in Imaging , 2016, IEEE Transactions on Image Processing.

[34]  Thomas B. Moeslund,et al.  Super-resolution: a comprehensive survey , 2014, Machine Vision and Applications.

[35]  Nikolas P. Galatsanos,et al.  Maximum a Posteriori Video Super-Resolution Using a New Multichannel Image Prior , 2010, IEEE Transactions on Image Processing.

[36]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[37]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Tong Tong,et al.  Image Super-Resolution Using Dense Skip Connections , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  Stephan Saalfeld,et al.  Deep Learning for Isotropic Super-Resolution from Non-isotropic 3D Electron Microscopy , 2017, MICCAI.

[40]  Timo Aila,et al.  Pruning Convolutional Neural Networks for Resource Efficient Inference , 2016, ICLR.

[41]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[42]  Yang Hu,et al.  Endoscopic Depth Measurement and Super-Spectral-Resolution Imaging , 2017, MICCAI.

[43]  Feng Shi,et al.  Brain MRI super resolution using 3D deep densely connected neural networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[44]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[48]  Konstantinos Kamnitsas,et al.  Anatomically Constrained Neural Networks (ACNNs): Application to Cardiac Image Enhancement and Segmentation , 2017, IEEE Transactions on Medical Imaging.

[49]  Zheng Zhang,et al.  MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems , 2015, ArXiv.

[50]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[51]  Lingfeng Wang,et al.  Image super-resolution via deep dilated convolutional networks , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[52]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[53]  Kwang In Kim,et al.  On Implicit Filter Level Sparsity in Convolutional Neural Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[55]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[56]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[57]  Tao Zhang,et al.  Channel Splitting Network for Single MR Image Super-Resolution , 2018, IEEE Transactions on Image Processing.

[58]  Pourya Shamsolmoali,et al.  Image super resolution by dilated dense progressive network , 2019, Image Vis. Comput..

[59]  Guang Li,et al.  CT Super-Resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble (GAN-CIRCLE) , 2018, IEEE Transactions on Medical Imaging.

[60]  Dwarikanath Mahapatra,et al.  Retinal Vasculature Segmentation Using Local Saliency Maps and Generative Adversarial Networks For Image Super Resolution , 2017, ArXiv.

[61]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[62]  Yu Yang,et al.  Simultaneous single- and multi-contrast super-resolution for brain MRI images based on a convolutional neural network , 2018, Comput. Biol. Medicine.

[63]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[64]  Pourya Shamsolmoali,et al.  G-GANISR: Gradual generative adversarial network for image super resolution , 2019, Neurocomputing.

[65]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[66]  Jae Sung Lee,et al.  Computed tomography super-resolution using deep convolutional neural network , 2018, Physics in medicine and biology.

[67]  Alan C. Bovik,et al.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms , 2006, IEEE Transactions on Image Processing.

[68]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[69]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[70]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[71]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[72]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  Aline Roumy,et al.  Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding , 2012, BMVC.

[74]  Dwarikanath Mahapatra,et al.  Image super-resolution using progressive generative adversarial networks for medical image analysis , 2019, Comput. Medical Imaging Graph..

[75]  Xuanqin Mou,et al.  Low-Dose CT Image Denoising Using a Generative Adversarial Network With Wasserstein Distance and Perceptual Loss , 2017, IEEE Transactions on Medical Imaging.

[76]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[77]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[78]  Marius George Linguraru,et al.  Adversarial approach to diagnostic quality volumetric image enhancement , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[79]  Li Xu,et al.  Shepard Convolutional Neural Networks , 2015, NIPS.

[80]  Michael Unser,et al.  Convolutional Neural Networks for Inverse Problems in Imaging: A Review , 2017, IEEE Signal Processing Magazine.

[81]  Verónica Vilaplana,et al.  Brain MRI super-resolution using 3D generative adversarial networks , 2018, ArXiv.

[82]  Jian Zhang,et al.  Fusing multi-scale information in convolution network for MR image super-resolution reconstruction , 2018, BioMedical Engineering OnLine.

[83]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[84]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..