Synthesis of diagnostic quality cancer pathology images by generative adversarial networks

Deep learning‐based computer vision methods have recently made remarkable breakthroughs in the analysis and classification of cancer pathology images. However, there has been relatively little investigation of the utility of deep neural networks to synthesize medical images. In this study, we evaluated the efficacy of generative adversarial networks to synthesize high‐resolution pathology images of 10 histological types of cancer, including five cancer types from The Cancer Genome Atlas and the five major histological subtypes of ovarian carcinoma. The quality of these images was assessed using a comprehensive survey of board‐certified pathologists (n = 9) and pathology trainees (n = 6). Our results show that the real and synthetic images are classified by histotype with comparable accuracies and the synthetic images are visually indistinguishable from real images. Furthermore, we trained deep convolutional neural networks to diagnose the different cancer types and determined that the synthetic images perform as well as additional real images when used to supplement a small training set. These findings have important applications in proficiency testing of medical practitioners and quality assurance in clinical laboratories. Furthermore, training of computer‐aided diagnostic systems can benefit from synthetic images where labeled datasets are limited (e.g. rare cancers). We have created a publicly available website where clinicians and researchers can attempt questions from the image survey (http://gan.aimlab.ca/). © 2020 Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.

[1]  Thomas J. Fuchs,et al.  Clinical-grade computational pathology using weakly supervised deep learning on whole slide images , 2019, Nature Medicine.

[2]  M. McHugh Interrater reliability: the kappa statistic , 2012, Biochemia medica.

[3]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[4]  Aleksey Boyko,et al.  Detecting Cancer Metastases on Gigapixel Pathology Images , 2017, ArXiv.

[5]  Heinz Handels,et al.  Multi-scale GANs for Memory-efficient Generation of High Resolution Medical Images , 2019, MICCAI.

[6]  Timo Kohlberger,et al.  Microscope 2.0: An Augmented Reality Microscope with Real-time Artificial Intelligence Integration , 2018, ArXiv.

[7]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[8]  Shin'ichi Satoh,et al.  Learning More with Less: GAN-based Medical Image Augmentation , 2019, ArXiv.

[9]  Jakob Nikolas Kather,et al.  Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer , 2019, Nature Medicine.

[10]  Rishi Sharma,et al.  A Note on the Inception Score , 2018, ArXiv.

[11]  Carl Doersch,et al.  Tutorial on Variational Autoencoders , 2016, ArXiv.

[12]  Andrew H. Beck,et al.  Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer , 2017, JAMA.

[13]  Léon Bottou,et al.  Towards Principled Methods for Training Generative Adversarial Networks , 2017, ICLR.

[14]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[15]  David F. Steiner,et al.  Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer , 2018, The American journal of surgical pathology.

[16]  N. Razavian,et al.  Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning , 2018, Nature Medicine.

[17]  Roderick Murray-Smith,et al.  Pathology GAN: Learning deep representations of cancer tissue , 2019, MIDL.

[18]  Francesmary Modugno,et al.  Biomarker-Based Ovarian Carcinoma Typing: A Histologic Investigation in the Ovarian Tumor Tissue Analysis Consortium , 2013, Cancer Epidemiology, Biomarkers & Prevention.

[19]  Timo Kohlberger,et al.  An augmented reality microscope with real-time artificial intelligence integration for cancer diagnosis , 2019, Nature Medicine.

[20]  Samuel Leung,et al.  Diagnosis of Ovarian Carcinoma Cell Type is Highly Reproducible: A Transcanadian Study , 2010, The American journal of surgical pathology.

[21]  Allan Donner,et al.  A confidence interval approach to sample size estimation for interobserver agreement studies with multiple raters and outcomes. , 2012, Journal of clinical epidemiology.

[22]  J. Fleiss Measuring nominal scale agreement among many raters. , 1971 .

[23]  Yoshua Bengio,et al.  Generative Adversarial Networks , 2014, ArXiv.

[24]  G. Wainrib,et al.  Deep learning-based classification of mesothelioma improves prediction of patient outcome , 2019, Nature Medicine.

[25]  Paul Babyn,et al.  Generative Adversarial Network in Medical Imaging: A Review , 2018, Medical Image Anal..

[26]  L. Leape,et al.  Problem Doctors: Is There a System-Level Solution? , 2006, Annals of Internal Medicine.

[27]  Jie Xu,et al.  The practical implementation of artificial intelligence technologies in medicine , 2019, Nature Medicine.

[28]  Qianni Zhang,et al.  GAN-based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis , 2019, ArXiv.

[29]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[30]  William C McGaghie,et al.  Medical Education Research As Translational Science , 2010, Science Translational Medicine.

[31]  S. Leung,et al.  Ovarian Carcinoma Subtypes Are Different Diseases: Implications for Biomarker Studies , 2008, PLoS medicine.

[32]  Cynthia Dwork,et al.  Differential Privacy: A Survey of Results , 2008, TAMC.

[33]  Robin Coope,et al.  Rise of the Machines: Advances in Deep Learning for Cancer Diagnosis. , 2019, Trends in cancer.

[34]  Yuan Liu,et al.  DermGAN: Synthetic Generation of Clinical Skin Images with Pathology , 2019, ML4H@NeurIPS.

[35]  W. Price,et al.  Privacy in the age of medical big data , 2019, Nature Medicine.

[36]  Georg Langs,et al.  Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery , 2017, IPMI.

[37]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[38]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[39]  A. Evans,et al.  Implementation of Whole Slide Imaging for Clinical Purposes: Issues to Consider From the Perspective of Early Adopters. , 2017, Archives of pathology & laboratory medicine.

[40]  Hayit Greenspan,et al.  GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification , 2018, Neurocomputing.

[41]  Leon A. Gatys,et al.  Texture Synthesis Using Convolutional Neural Networks , 2015, NIPS.

[42]  Neil J. Joshi,et al.  Assessment of Deep Generative Models for High-Resolution Synthetic Retinal Image Generation of Age-Related Macular Degeneration , 2019, JAMA ophthalmology.

[43]  Joel H. Saltz,et al.  Unsupervised Histopathology Image Synthesis , 2017, ArXiv.

[44]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[45]  Sameer Antani,et al.  Synthetic Augmentation and Feature-based Filtering for Improved Cervical Histopathology Image Classification , 2019, MICCAI.

[46]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[47]  Nassir Navab,et al.  Staingan: Stain Style Transfer for Digital Histological Images , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[48]  Ellery Wulczyn,et al.  Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer , 2018, npj Digital Medicine.

[49]  D. Brat,et al.  Predicting cancer outcomes from histology and genomics using convolutional networks , 2017, Proceedings of the National Academy of Sciences.

[50]  Richard J. Chen,et al.  Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images , 2018, IEEE Transactions on Medical Imaging.

[51]  Stefan Kommoss,et al.  Ovarian carcinoma diagnosis: the clinical impact of 15 years of change , 2016, British Journal of Cancer.

[52]  Allison P. Heath,et al.  Toward a Shared Vision for Cancer Genomic Data. , 2016, The New England journal of medicine.

[53]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.