Unsupervised MRI Super Resolution Using Deep External Learning and Guided Residual Dense Network With Multimodal Image Priors

Deep learning techniques have led to state-of-the-art single image super-resolution (SISR) with natural images. Pairs of high-resolution (HR) and low-resolution (LR) images are used to train the deep learning model (mapping function). These techniques have also been applied to medical image super-resolution (SR). Compared with natural images, medical images have several unique characteristics. First, there are no HR images for training in real clinical applications because of the limitations of imaging systems and clinical requirements. Second, other modal HR images are available (e.g., HR T1-weighted images are available for enhancing LR T2-weighted images). In this paper, we propose an unsupervised SISR technique based on simple prior knowledge of the human anatomy; this technique does not require HR images for training. Furthermore, we present a guided residual dense network, which incorporates a residual dense network with a guided deep convolutional neural network for enhancing the resolution of LR images by referring to different HR images of the same subject. Experiments on a publicly available brain MRI database showed that our proposed method achieves better performance than the state-of-the-art methods.

[1]  Yen-Wei Chen,et al.  Fast super-resolution with iterative-guided back projection for 3D MR images , 2018, Medical Imaging.

[2]  Luc Van Gool,et al.  Seven Ways to Improve Example-Based Single Image Super Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Liangpei Zhang,et al.  Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[4]  Michal Irani,et al.  "Zero-Shot" Super-Resolution Using Deep Internal Learning , 2017, CVPR.

[5]  Nicolas Passat,et al.  Multiscale brain MRI super-resolution using deep 3D convolutional networks , 2019, Comput. Medical Imaging Graph..

[6]  Amod Jog,et al.  Self Super-Resolution for Magnetic Resonance Images , 2016, MICCAI.

[7]  Hayit Greenspan,et al.  MRI inter-slice reconstruction using super-resolution , 2002 .

[8]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[10]  Debiao Li,et al.  Efficient and Accurate MRI Super-Resolution using a Generative Adversarial Network and 3D Multi-Level Densely Connected Network , 2018, MICCAI.

[11]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[12]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  Huazhu Fu,et al.  Hierarchical Features Driven Residual Learning for Depth Map Super-Resolution , 2019, IEEE Transactions on Image Processing.

[14]  Yutaro Iwamoto,et al.  Color Guided Depth Map Super-Resolution based on a Deep Self-Learning Approach , 2020, 2020 IEEE International Conference on Consumer Electronics (ICCE).

[15]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[17]  Feng Shi,et al.  Brain MRI super resolution using 3D deep densely connected neural networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[18]  D. Louis Collins,et al.  Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging , 2013, Medical Image Anal..

[19]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Wilfried Philips,et al.  MRI Segmentation of the Human Brain: Challenges, Methods, and Applications , 2015, Comput. Math. Methods Medicine.

[21]  Yen-Wei Chen,et al.  Super-resolution of MR volumetric images using sparse representation and self-similarity , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[22]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Nelly Gordillo,et al.  State of the art survey on MRI brain tumor segmentation. , 2013, Magnetic resonance imaging.

[24]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Aaron Carass,et al.  Self super-resolution for magnetic resonance images using deep networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[26]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[27]  D. Louis Collins,et al.  MRI Superresolution Using Self-Similarity and Image Priors , 2010, Int. J. Biomed. Imaging.

[28]  Hayit Greenspan,et al.  MRI Inter-slice Reconstruction Using Super-Resolution , 2001, MICCAI.

[29]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[30]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[31]  François Rousseau,et al.  A non-local approach for image super-resolution using intermodality priors , 2010, Medical Image Anal..