PathSRGAN: Multi-Supervised Super-Resolution for Cytopathological Images Using Generative Adversarial Network

In the cytopathology screening of cervical cancer, high-resolution digital cytopathological slides are critical for the interpretation of lesion cells. However, the acquisition of high-resolution digital slides requires high-end imaging equipment and long scanning time. In the study, we propose a GAN-based progressive multi-supervised super-resolution model called PathSRGAN (pathology super-resolution GAN) to learn the mapping of real low-resolution and high-resolution cytopathological images. With respect to the characteristics of cytopathological images, we design a new two-stage generator architecture with two supervision terms. The generator of the first stage corresponds to a densely-connected U-Net and achieves <inline-formula> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> to <inline-formula> <tex-math notation="LaTeX">$10\times $ </tex-math></inline-formula> super resolution. The generator of the second stage corresponds to a residual-in-residual DenseBlock and achieves <inline-formula> <tex-math notation="LaTeX">$10\times $ </tex-math></inline-formula> to <inline-formula> <tex-math notation="LaTeX">$20\times $ </tex-math></inline-formula> super resolution. The designed generator alleviates the difficulty in learning the mapping from <inline-formula> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> images to <inline-formula> <tex-math notation="LaTeX">$20\times $ </tex-math></inline-formula> images caused by the great numerical aperture difference and generates high quality high-resolution images. We conduct a series of comparison experiments and demonstrate the superiority of PathSRGAN to mainstream CNN-based and GAN-based super-resolution methods in cytopathological images. Simultaneously, the reconstructed high-resolution images by PathSRGAN improve the accuracy of computer-aided diagnosis tasks effectively. It is anticipated that the study will help increase the penetration rate of cytopathology screening in remote and impoverished areas that lack high-end imaging equipment.

[1]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[3]  Robert J. Eckersley,et al.  In Vivo Acoustic Super-Resolution and Super-Resolved Velocity Mapping Using Microbubbles , 2015, IEEE Transactions on Medical Imaging.

[4]  C. Duchon.  Lanczos Filtering in One and Two Dimensions , 1979 .

[5]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[6]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[7]  A. Ozcan,et al.  Deep learning enables cross-modality super-resolution in fluorescence microscopy , 2018, Nature Methods.

[8]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[9]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[11]  Ronald M. Summers,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, IEEE Transactions on Medical Imaging.

[12]  Alexia Jolicoeur-Martineau,et al.  The relativistic discriminator: a key element missing from standard GAN , 2018, ICLR.

[13]  Ronald M. Summers,et al.  Improving Computer-Aided Detection Using Convolutional Neural Networks and Random View Aggregation , 2015, IEEE Transactions on Medical Imaging.

[14]  Kyung-Ah Sohn,et al.  Fast, Accurate, and, Lightweight Super-Resolution with Cascading Residual Network , 2018, ECCV.

[15]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[16]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Horst Bischof,et al.  Fast and accurate image upscaling with super-resolution forests , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Hong Chang,et al.  Super-resolution through neighbor embedding , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[19]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[20]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Kwang In Kim,et al.  Single-Image Super-Resolution Using Sparse Regression and Natural Image Prior , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Yibo Zhang,et al.  Deep Learning Microscopy , 2017, ArXiv.

[23]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Georgios Tzimiropoulos,et al.  Super-FAN: Integrated Facial Landmark Localization and Super-Resolution of Real-World Low Resolution Faces in Arbitrary Poses with GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[26]  D. Shen,et al.  LRTV: MR Image Super-Resolution With Low-Rank and Total Variation Regularizations , 2015, IEEE Transactions on Medical Imaging.

[27]  Quan Pan,et al.  Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Yifan Wang,et al.  A Fully Progressive Approach to Single-Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[30]  Bernhard Schölkopf,et al.  EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  C. Dunsby,et al.  3-D In Vitro Acoustic Super-Resolution and Super-Resolved Velocity Mapping Using Microbubbles , 2015, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[32]  Peter Bankhead,et al.  QuPath: Open source software for digital pathology image analysis , 2017, Scientific Reports.

[33]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Guang Li,et al.  CT Super-Resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble (GAN-CIRCLE) , 2018, IEEE Transactions on Medical Imaging.

[35]  Siyuan Liu,et al.  Unsupervised Image Super-Resolution Using Cycle-in-Cycle Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[36]  Peng Fei,et al.  High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network. , 2019, Biomedical optics express.