Generative Adversarial Networks in Digital Pathology: A Survey on Trends and Future Potential

Summary Image analysis in the field of digital pathology has recently gained increased popularity. The use of high-quality whole-slide scanners enables the fast acquisition of large amounts of image data, showing extensive context and microscopic detail at the same time. Simultaneously, novel machine-learning algorithms have boosted the performance of image analysis approaches. In this paper, we focus on a particularly powerful class of architectures, the so-called generative adversarial networks (GANs) applied to histological image data. Besides improving performance, GANs also enable previously intractable application scenarios in this field. However, GANs could exhibit a potential for introducing bias. Hereby, we summarize the recent state-of-the-art developments in a generalizing notation, present the main applications of GANs, and give an outlook of some chosen promising approaches and their possible future applications. In addition, we identify currently unavailable methods with potential for future applications.

[1]  D. Salmi,et al.  Trends in Pathology Graduate Medical Education Programs and Positions, 2001 to 2017 , 2018, Academic pathology.

[2]  Thomas Mensink,et al.  Image Classification with the Fisher Vector: Theory and Practice , 2013, International Journal of Computer Vision.

[3]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Neofytos Dimitriou,et al.  Deep Learning for Whole Slide Image Analysis: An Overview , 2019, Front. Med..

[5]  Honglak Lee,et al.  Object Contour Detection with a Fully Convolutional Encoder-Decoder Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Yiping Wang,et al.  Synthesis of diagnostic quality cancer pathology images , 2020, bioRxiv.

[7]  Yuan Liu,et al.  DermGAN: Synthetic Generation of Clinical Skin Images with Pathology , 2019, ML4H@NeurIPS.

[8]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[9]  Xin Qi,et al.  Adversarial Domain Adaptation for Classification of Prostate Histopathology Whole-Slide Images , 2018, MICCAI.

[10]  J. S. Marron,et al.  A method for normalizing histology slides for quantitative analysis , 2009, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[11]  Mateusz Buda,et al.  MRI image harmonization using cycle-consistent generative adversarial network , 2020, Medical Imaging.

[12]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Joel H. Saltz,et al.  Ieee Transactions on Visualization and Computer Graphics Reconstruction of Cellular Biological Structures from Optical Microscopy Data , 2022 .

[14]  Linda G. Shapiro,et al.  Detection and classification of cancer in whole slide breast histopathology images using deep convolutional networks , 2018, Pattern Recognit..

[15]  Nassir Navab,et al.  Perceptual Embedding Consistency for Seamless Reconstruction of Tilewise Style Transfer , 2019, MICCAI.

[16]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[17]  Saeed Hassanpour,et al.  Generative Image Translation for Data Augmentation in Colorectal Histopathology Images , 2019, ML4H@NeurIPS.

[18]  Yang Lei,et al.  MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks. , 2019, Medical physics.

[19]  Berkman Sahiner,et al.  Optimized generation of high-resolution phantom images using cGAN: Application to quantification of Ki67 breast cancer images , 2018, PloS one.

[20]  Dorit Merhof,et al.  Supervised and Unsupervised Cell-Nuclei Detection in Immunohistology , 2019 .

[21]  Mahmoud Abdolhoseini,et al.  Segmentation of Heavily Clustered Nuclei from Histopathological Images , 2019, Scientific Reports.

[22]  Peter H. N. de With,et al.  Stain normalization of histopathology images using generative adversarial networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[23]  Michael Gadermayr,et al.  An Asymetric Cycle-Consistency Loss for Dealing with Many-to-One Mappings in Image Translation: A Study on Thigh MR Scans , 2020, ArXiv.

[24]  Marek Kowal,et al.  Cell Nuclei Segmentation in Cytological Images Using Convolutional Neural Network and Seeded Watershed Algorithm , 2019, Journal of Digital Imaging.

[25]  Hao Chen,et al.  Gland segmentation in colon histology images: The glas challenge contest , 2016, Medical Image Anal..

[26]  Nico Karssemeijer,et al.  Large scale deep learning for computer aided detection of mammographic lesions , 2017, Medical Image Anal..

[27]  Xin-Ping Guan,et al.  Adversarial neural networks for basal membrane segmentation of microinvasive cervix carcinoma in histopathology images , 2017, 2017 International Conference on Machine Learning and Cybernetics (ICMLC).

[28]  Michael Gadermayr,et al.  CNN cascades for segmenting sparse objects in gigapixel whole slide images , 2019, Comput. Medical Imaging Graph..

[29]  Pieter Abbeel,et al.  InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets , 2016, NIPS.

[30]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[31]  Joel H. Saltz,et al.  Patch-Based Convolutional Neural Network for Whole Slide Tissue Image Classification , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Richard J. Chen,et al.  Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images , 2018, IEEE Transactions on Medical Imaging.

[33]  Yiping Wang,et al.  Synthesis of diagnostic quality cancer pathology images by generative adversarial networks , 2020, The Journal of pathology.

[34]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[35]  Nasir M. Rajpoot,et al.  A Nonlinear Mapping Approach to Stain Normalization in Digital Histopathology Images Using Image-Specific Color Deconvolution , 2014, IEEE Transactions on Biomedical Engineering.

[36]  Tomas Pfister,et al.  Learning from Simulated and Unsupervised Images through Adversarial Training , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[38]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[39]  Jelmer M. Wolterink,et al.  Deep MR to CT Synthesis Using Unpaired Data , 2017, SASHIMI@MICCAI.

[40]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[41]  Qianni Zhang,et al.  GAN-based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis , 2019, ArXiv.

[42]  Ghassan Hamarneh,et al.  Topology Aware Fully Convolutional Networks for Histology Gland Segmentation , 2016, MICCAI.

[43]  Tadashi Hasegawa,et al.  Evaluation of the Interobserver Agreement in the Number of Mitotic Figures Breast Carcinoma as Simulation of Quality Monitoring in the Japan National Surgical Adjuvant Study of Breast Cancer (NSAS‐BC) Protocol , 2000, Japanese journal of cancer research : Gann.

[44]  Geert J. S. Litjens,et al.  Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology , 2019, Medical Image Anal..

[45]  Jianhua Yao,et al.  Enhanced Cycle-Consistent Generative Adversarial Network for Color Normalization of H&E Stained Images , 2019, MICCAI.

[46]  Timothy D. Solberg,et al.  Attention-Aware Discrimination for MR-to-CT Image Translation Using Cycle-Consistent Generative Adversarial Networks. , 2020, Radiology. Artificial intelligence.

[47]  Philip Bachman,et al.  Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data , 2018, ICML.

[48]  Roderick Murray-Smith,et al.  Pathology GAN: Learning deep representations of cancer tissue , 2019, MIDL.

[49]  Geert J. S. Litjens,et al.  Stain-Transforming Cycle-Consistent Generative Adversarial Networks for Improved Segmentation of Renal Histopathology , 2018, MIDL.

[50]  Gregory R. Koch,et al.  Siamese Neural Networks for One-Shot Image Recognition , 2015 .

[51]  Ghassan Hamarneh,et al.  Adversarial Stain Transfer for Histopathology Image Analysis , 2018, IEEE Transactions on Medical Imaging.

[52]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[53]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[54]  Joseph O. Deasy,et al.  Tumor-Aware, Adversarial Domain Adaptation from CT to MRI for Lung Cancer Segmentation , 2018, MICCAI.

[55]  Shunxing Bao,et al.  Adversarial synthesis learning enables segmentation without target modality ground truth , 2017, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[56]  James M. Brown,et al.  Semi-Supervised Deep Learning for Abnormality Classification in Retinal Images , 2018, ArXiv.

[57]  Michael Gadermayr,et al.  Generative Adversarial Networks for Facilitating Stain-Independent Supervised and Unsupervised Segmentation: A Study on Kidney Histology , 2019, IEEE Transactions on Medical Imaging.

[58]  Michael Gadermayr,et al.  Domain Adaptive Classification for Compensating Variability in Histopathological Whole Slide Images , 2016, ICIAR.

[59]  Abhinav Gupta,et al.  Training Region-Based Object Detectors with Online Hard Example Mining , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Gunnar Steineck,et al.  Interobserver variability in the pathological assessment of radical prostatectomy specimens: Findings of the Laparoscopic Prostatectomy Robot Open (LAPPRO) study , 2014, Scandinavian journal of urology.

[61]  Anant Gupta,et al.  Generative Image Translation for Data Augmentation of Bone Lesion Pathology , 2018, MIDL.

[62]  Michael Gadermayr,et al.  Which Way Round? A Study on the Performance of Stain-Translation for Segmenting Arbitrarily Dyed Histological Images , 2018, MICCAI.

[63]  Nassir Navab,et al.  Staingan: Stain Style Transfer for Digital Histological Images , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[64]  Joel H. Saltz,et al.  Unsupervised Histopathology Image Synthesis , 2017, ArXiv.

[65]  Zhuowen Tu,et al.  Weakly supervised histopathology cancer image segmentation and classification , 2014, Medical Image Anal..

[66]  Joel H. Saltz,et al.  Robust Histopathology Image Analysis: To Label or to Synthesize? , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  Frédo Durand,et al.  Data augmentation using learned transforms for one-shot medical image segmentation , 2019, ArXiv.

[68]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[69]  Jason Y. Park,et al.  Trends in the US and Canadian Pathologist Workforces From 2007 to 2017 , 2019, JAMA network open.

[70]  Michael Gadermayr,et al.  GAN-Based Image Enrichment in Digital Pathology Boosts Segmentation Accuracy , 2019, MICCAI.

[71]  Aman Rana,et al.  Computational Histological Staining and Destaining of Prostate Core Biopsy RGB Images with Generative Adversarial Neural Networks , 2018, 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA).

[72]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[73]  Bo Hu,et al.  Unsupervised Learning for Cell-Level Visual Representation in Histopathology Images With Generative Adversarial Networks , 2017, IEEE Journal of Biomedical and Health Informatics.

[74]  Joe W. Gray,et al.  SHIFT: speedy histopathological-to-immunofluorescent translation of whole slide images using conditional generative adversarial networks , 2018, Medical Imaging.

[75]  Chanho Jung,et al.  Unsupervised Segmentation of Overlapped Nuclei Using Bayesian Classification , 2010, IEEE Transactions on Biomedical Engineering.

[76]  Michael Gadermayr,et al.  Unsupervisedly Training GANs for Segmenting Digital Pathology with Automatically Generated Annotations , 2018, MIDL.