Breast Cancer Histopathology Image Super-Resolution Using Wide-Attention GAN With Improved Wasserstein Gradient Penalty and Perceptual Loss

In the realm of image processing, enhancing the quality of the images is known as a super-resolution problem (SR). Among SR methods, a super-resolution generative adversarial network, or SRGAN, has been introduced to generate SR images from low-resolution images. As it is of the utmost importance to keep the size and the shape of the images, while enlarging the medical images, we propose a novel super-resolution model with a generative adversarial network to generate SR images with finer details and higher quality to encourage less blurring. By widening residual blocks and using a self-attention layer, our model becomes robust and generalizable as it is able to extract the most important part of the images before up-sampling. We named our proposed model as wide-attention SRGAN (WA-SRGAN). Moreover, we have applied improved Wasserstein with a Gradient penalty to stabilize the model while training. To train our model, we have applied images from Camylon 16 database and enlarged them by <inline-formula> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$8\times $ </tex-math></inline-formula>, and <inline-formula> <tex-math notation="LaTeX">$16\times $ </tex-math></inline-formula> upscale factors with the ground truth of the size of <inline-formula> <tex-math notation="LaTeX">$256\times 256\times 3$ </tex-math></inline-formula>. Furthermore, two normalization methods, including batch normalization, and weight normalization have been applied and we observed that weight normalization is an enabling factor to improve metric performance in terms of SSIM. Moreover, several evaluation metrics, such as PSNR, MSE, SSIM, MS-SSIM, and QILV have been applied for having a comprehensive objective comparison with other methods, including SRGAN, A-SRGAN, and bicubial. Also, we performed the job of classification by using a deep learning model called ResNeXt-101 (<inline-formula> <tex-math notation="LaTeX">$32\times 8\text{d}$ </tex-math></inline-formula>) for super-resolution, high-resolution, and low-resolution images and compared the outcomes in terms of accuracy score. Finally, the results on breast cancer histopathology images show the superiority of our model by using weight normalization and a batch size of one in terms of restoration of the color and the texture details.

[1]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[3]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[4]  Yi Li,et al.  Cancer Metastasis Detection With Neural Conditional Random Field , 2018, ArXiv.

[5]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[6]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Sitao Xiang,et al.  On the Effects of Batch and Weight Normalization in Generative Adversarial Networks , 2017 .

[8]  Boris Ginsburg,et al.  Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification , 2017, ArXiv.

[9]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[10]  Guoli Wang,et al.  Medical Image Super-Resolution Based on the Generative Adversarial Network , 2019, Lecture Notes in Electrical Engineering.

[11]  Suvrit Sra,et al.  Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity , 2019, ICLR.

[12]  Alex Graves,et al.  Conditional Image Generation with PixelCNN Decoders , 2016, NIPS.

[13]  Luiz Eduardo Soares de Oliveira,et al.  A Dataset for Breast Cancer Histopathological Image Classification , 2016, IEEE Transactions on Biomedical Engineering.

[14]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[15]  Ning Xu,et al.  Wide Activation for Efficient and Accurate Image Super-Resolution , 2018, ArXiv.

[16]  Suyash P. Awate,et al.  Robust Super-Resolution Gan, with Manifold-Based and Perception Loss , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[17]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Bei Jiang,et al.  Negative Log Likelihood Ratio Loss for Deep Neural Network Classification , 2018, Advances in Intelligent Systems and Computing.

[19]  Bülent Sankur,et al.  Statistical evaluation of image quality measures , 2002, J. Electronic Imaging.

[20]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Lopamudra Mukherjee,et al.  Convolutional neural networks for whole slide image superresolution. , 2018, Biomedical optics express.

[22]  Ian J. Goodfellow,et al.  NIPS 2016 Tutorial: Generative Adversarial Networks , 2016, ArXiv.

[23]  Tim Salimans,et al.  Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks , 2016, NIPS.

[24]  R. Keys.  Cubic convolution interpolation for digital image processing , 1981 .

[25]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[26]  H. Ramesh,et al.  Effectiveness of Contrast Limited Adaptive Histogram Equalization Technique on Multispectral Satellite Imagery , 2017, ICVIP.

[27]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[28]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[29]  Muhammed Fatih Talu,et al.  Resizing and cleaning of histopathological images using generative adversarial networks , 2020 .

[30]  Timo Kohlberger,et al.  An augmented reality microscope with real-time artificial intelligence integration for cancer diagnosis , 2019, Nature Medicine.

[31]  Hong Liu,et al.  A Fast and Refined Cancer Regions Segmentation Framework in Whole-slide Breast Pathological Images , 2019, Scientific Reports.

[32]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[33]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[34]  C. Duchon.  Lanczos Filtering in One and Two Dimensions , 1979 .

[35]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[36]  Thomas S. Huang,et al.  Deep Networks for Image Super-Resolution with Sparse Prior , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[37]  Ashlesha Vaidya,et al.  Batch Normalization and Its Optimization Techniques: Review , 2017 .

[38]  Noor Azurati Ahmad,et al.  Breast Cancer Classification Using Deep Learning Approaches and Histopathology Image: A Comparison Study , 2020, IEEE Access.

[39]  Raúl San José Estépar,et al.  Image Quality Assessment based on Local Variance , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[40]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[41]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[42]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[43]  Shervin Minaee,et al.  Efficient Super Resolution for Large-Scale Images Using Attentional GAN , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[44]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[45]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[46]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[47]  Iman Hajirasouliha,et al.  Breast Cancer Histopathological Image Classification: A Deep Learning Approach , 2018, bioRxiv.

[48]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[49]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Hongyu Wang,et al.  End-to-End Image Super-Resolution via Deep and Shallow Convolutional Networks , 2016, IEEE Access.

[51]  James A. Ferwerda,et al.  Three varieties of realism in computer graphics , 2003, IS&T/SPIE Electronic Imaging.

[52]  Pascal Vincent,et al.  An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family , 2015, ICLR.

[53]  Djemel Ziou,et al.  Image Quality Metrics: PSNR vs. SSIM , 2010, 2010 20th International Conference on Pattern Recognition.

[54]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).