Synthesis of diagnostic quality cancer pathology images

Deep learning-based computer vision methods have recently made remarkable breakthroughs in the analysis and classification of cancer pathology images. However, there has been relatively little investigation of the utility of deep neural networks to synthesize medical images. In this study, we evaluated the efficacy of generative adversarial networks (GANs) to synthesize high resolution pathology images of ten histological types of cancer, including five cancer types from The Cancer Genome Atlas (TCGA) and the five major histological subtypes of ovarian carcinoma. The quality of these images was assessed using a comprehensive survey of board-certified pathologists (n = 9) and pathology trainees (n = 6). Our results show that the real and synthetic images are classified by histotype with comparable accuracies, and the synthetic images are visually indistinguishable from real images. Furthermore, we trained deep convolutional neural networks (CNNs) to diagnose the different cancer types and determined that the synthetic images perform as well as additional real images when used to supplement a small training set. These findings have important applications in proficiency testing of medical practitioners and quality assurance in clinical laboratories. Furthermore, training of computer-aided diagnostic systems can benefit from synthetic images where labeled datasets are limited (e.g., rare cancers). We have created a publicly available website where clinicians and researchers can attempt questions from the image survey at http://gan.aimlab.ca/.

[1]  Andrew H. Beck,et al.  Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer , 2017, JAMA.

[2]  David F. Steiner,et al.  Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer , 2018, The American journal of surgical pathology.

[3]  Sameer Antani,et al.  Synthetic Augmentation and Feature-based Filtering for Improved Cervical Histopathology Image Classification , 2019, MICCAI.

[4]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[5]  Jakob Nikolas Kather,et al.  Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer , 2019, Nature Medicine.

[6]  Léon Bottou,et al.  Towards Principled Methods for Training Generative Adversarial Networks , 2017, ICLR.

[7]  W. Price,et al.  Privacy in the age of medical big data , 2019, Nature Medicine.

[8]  S. Leung,et al.  Ovarian Carcinoma Subtypes Are Different Diseases: Implications for Biomarker Studies , 2008, PLoS medicine.

[9]  Cynthia Dwork,et al.  Differential Privacy: A Survey of Results , 2008, TAMC.

[10]  Allan Donner,et al.  A confidence interval approach to sample size estimation for interobserver agreement studies with multiple raters and outcomes. , 2012, Journal of clinical epidemiology.

[11]  N. Razavian,et al.  Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning , 2018, Nature Medicine.

[12]  Timo Kohlberger,et al.  An augmented reality microscope with real-time artificial intelligence integration for cancer diagnosis , 2019, Nature Medicine.

[13]  Samuel Leung,et al.  Diagnosis of Ovarian Carcinoma Cell Type is Highly Reproducible: A Transcanadian Study , 2010, The American journal of surgical pathology.

[14]  A. Evans,et al.  Implementation of Whole Slide Imaging for Clinical Purposes: Issues to Consider From the Perspective of Early Adopters. , 2017, Archives of pathology & laboratory medicine.

[15]  Carl Doersch,et al.  Tutorial on Variational Autoencoders , 2016, ArXiv.

[16]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[17]  Heinz Handels,et al.  Multi-scale GANs for Memory-efficient Generation of High Resolution Medical Images , 2019, MICCAI.

[18]  Hayit Greenspan,et al.  GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification , 2018, Neurocomputing.

[19]  Leon A. Gatys,et al.  Texture Synthesis Using Convolutional Neural Networks , 2015, NIPS.

[20]  Francesmary Modugno,et al.  Biomarker-Based Ovarian Carcinoma Typing: A Histologic Investigation in the Ovarian Tumor Tissue Analysis Consortium , 2013, Cancer Epidemiology, Biomarkers & Prevention.

[21]  D. Brat,et al.  Predicting cancer outcomes from histology and genomics using convolutional networks , 2017, Proceedings of the National Academy of Sciences.

[22]  G. Wainrib,et al.  Deep learning-based classification of mesothelioma improves prediction of patient outcome , 2019, Nature Medicine.

[23]  Paul Babyn,et al.  Generative Adversarial Network in Medical Imaging: A Review , 2018, Medical Image Anal..

[24]  Richard J. Chen,et al.  Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images , 2018, IEEE Transactions on Medical Imaging.

[25]  Thomas J. Fuchs,et al.  Clinical-grade computational pathology using weakly supervised deep learning on whole slide images , 2019, Nature Medicine.

[26]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Shin'ichi Satoh,et al.  Learning More with Less: Conditional PGGAN-based Data Augmentation for Brain Metastases Detection Using Highly-Rough Annotation on MR Images , 2019, CIKM.

[28]  Chen Wang,et al.  A Sensor Image Super-Resolution via Advanced Generative Adversarial Network , 2018 .

[29]  L. Leape,et al.  Problem Doctors: Is There a System-Level Solution? , 2006, Annals of Internal Medicine.

[30]  Shin'ichi Satoh,et al.  Learning More with Less: GAN-based Medical Image Augmentation , 2019, ArXiv.

[31]  Roderick Murray-Smith,et al.  Pathology GAN: Learning deep representations of cancer tissue , 2019, MIDL.

[32]  Stefan Kommoss,et al.  Ovarian carcinoma diagnosis: the clinical impact of 15 years of change , 2016, British Journal of Cancer.

[33]  Allison P. Heath,et al.  Toward a Shared Vision for Cancer Genomic Data. , 2016, The New England journal of medicine.

[34]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[35]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[36]  William C McGaghie,et al.  Medical Education Research As Translational Science , 2010, Science Translational Medicine.

[37]  Timo Kohlberger,et al.  Microscope 2.0: An Augmented Reality Microscope with Real-time Artificial Intelligence Integration , 2018, ArXiv.

[38]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[39]  Nassir Navab,et al.  Staingan: Stain Style Transfer for Digital Histological Images , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[40]  Ellery Wulczyn,et al.  Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer , 2018, npj Digital Medicine.

[41]  Jie Xu,et al.  The practical implementation of artificial intelligence technologies in medicine , 2019, Nature Medicine.

[42]  Robin Coope,et al.  Rise of the Machines: Advances in Deep Learning for Cancer Diagnosis. , 2019, Trends in cancer.

[43]  Yuan Liu,et al.  DermGAN: Synthetic Generation of Clinical Skin Images with Pathology , 2019, ML4H@NeurIPS.

[44]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[45]  Neil J. Joshi,et al.  Assessment of Deep Generative Models for High-Resolution Synthetic Retinal Image Generation of Age-Related Macular Degeneration , 2019, JAMA ophthalmology.

[46]  Joel H. Saltz,et al.  Unsupervised Histopathology Image Synthesis , 2017, ArXiv.

[47]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[48]  Aleksey Boyko,et al.  Detecting Cancer Metastases on Gigapixel Pathology Images , 2017, ArXiv.

[49]  Georg Langs,et al.  Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery , 2017, IPMI.

[50]  Qianni Zhang,et al.  GAN-based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis , 2019, ArXiv.