How Can We Make Gan Perform Better in Single Medical Image Super-Resolution? A Lesion Focused Multi-Scale Approach

Single image super-resolution (SISR) is of great importance as a low-level computer vision task. The fast development of Generative Adversarial Network (GAN) based deep learning architectures realises an efficient and effective SISR to boost the spatial resolution of natural images captured by digital cameras. However, the SISR for medical images is still a very challenging problem. This is due to (1) compared to natural images, in general, medical images have lower signal to noise ratios, (2) GAN based models pre-trained on natural images may synthesise unrealistic patterns in medical images which could affect the clinical interpretation and diagnosis, and (3) the vanilla GAN architecture may suffer from unstable training and collapse mode that can also affect the SISR results. In this paper, we propose a novel lesion focused SR (LFSR) method, which incorporates GAN to achieve perceptually realistic SISR results for brain tumour MRI images. More importantly, we test and make comparison using recently developed GAN variations, e.g., Wasserstein GAN (WGAN) and WGAN with Gradient Penalty (WGAN-GP), and propose a novel multi-scale GAN (MS-GAN), to achieve a more stabilised and efficient training and improved perceptual quality of the super-resolved results. Based on both quantitative evaluations and our designed mean opinion score, the proposed LFSR coupled with MS-GAN has performed better in terms of both perceptual quality and efficiency.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[3]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Debiao Li,et al.  Efficient and Accurate MRI Super-Resolution using a Generative Adversarial Network and 3D Multi-Level Densely Connected Network , 2018, MICCAI.

[5]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[7]  Liang Han,et al.  A Cascaded Refinement GAN for Phase Contrast Microscopy Image Super Resolution , 2018, MICCAI.

[8]  Chia-Hung Yeh,et al.  Self-learning-based single image super-resolution of a highly compressed image , 2013, 2013 IEEE 15th International Workshop on Multimedia Signal Processing (MMSP).

[9]  Guang Yang,et al.  Deep De-Aliasing for Fast Compressive Sensing MRI , 2017, ArXiv.

[10]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[11]  Guang Yang,et al.  Adversarial and Perceptual Refinement for Compressed Sensing MRI Reconstruction , 2018, MICCAI.

[12]  D. Yeung,et al.  Super-resolution through neighbor embedding , 2004, CVPR 2004.

[13]  Thomas S. Huang,et al.  Coupled Dictionary Training for Image Super-Resolution , 2012, IEEE Transactions on Image Processing.

[14]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Xinbo Gao,et al.  Fast and Accurate Single Image Super-Resolution via Information Distillation Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Guang Yang,et al.  Fully automatic segmentation and objective assessment of atrial scars for long‐standing persistent atrial fibrillation patients using late gadolinium‐enhanced MRI , 2017, Medical physics.

[17]  Narendra Ahuja,et al.  Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Truong Q. Nguyen,et al.  Novel Example-Based Method for Super-Resolution and Denoising of Medical Images , 2014, IEEE Transactions on Image Processing.

[19]  Guang Yang,et al.  Combined self-learning based single-image super-resolution and dual-tree complex wavelet transform denoising for medical images , 2016, SPIE Medical Imaging.

[20]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Guang Yang,et al.  Super-Resolved Enhancement of a Single Image and Its Application in Cardiac MRI , 2016, ICISP.

[22]  Guang Yang,et al.  Automatic Brain Tumor Detection and Segmentation Using U-Net Based Fully Convolutional Networks , 2017, MIUA.

[23]  Xin Deng Enhancing Image Quality via Style Transfer for Single Image Super-Resolution , 2018, IEEE Signal Processing Letters.

[24]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  W. Moses Fundamental Limits of Spatial Resolution in PET. , 2011, Nuclear instruments & methods in physics research. Section A, Accelerators, spectrometers, detectors and associated equipment.

[26]  Guang Yang,et al.  DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction , 2018, IEEE Transactions on Medical Imaging.

[27]  Chih-Yuan Yang,et al.  Exploiting Self-similarities for Single Frame Super-Resolution , 2010, ACCV.