A Large-Scale Internal Validation Study of Unsupervised Virtual Trichrome Staining Technologies on Non-alcoholic Steatohepatitis Liver Biopsies

Non-alcoholic steatohepatitis (NASH) is a fatty liver disease characterized by accumulation of fat in hepatocytes with concurrent inflammation and is associated with morbidity, cirrhosis and liver failure. After extraction of a liver core biopsy, tissue sections are stained with hematoxylin and eosin (H&E) to grade NASH activity, and stained with trichrome to stage fibrosis. Methods to computationally transform one stain into another on digital whole slide images (WSI) can lessen the need for additional physical staining besides H&E, reducing personnel, equipment, and time costs. Generative adversarial networks (GAN) have shown promise for virtual staining of tissue. We conducted a large-scale validation study of the viability of GANs for H&E to trichrome conversion on WSI (n=574). Pathologists were largely unable to distinguish real images from virtual/synthetic images given a set of twelve Turing Tests. We report high correlation between staging of real and virtual stains (ρ = 0.86; 95% CI: 0.84-0.88). Stages assigned to both virtual and real stains correlated similarly with a number of clinical biomarkers and progression to End Stage Liver Disease (Hazard Ratio HR = 2.06, CI 95% 1.36-3.12, P < 0.001 for real stains; HR = 2.02, CI 95% 1.40-2.92, p < 0.001 for virtual stains). Our results demonstrate that virtual trichrome technologies may offer a software solution that can be employed in the clinical setting as a diagnostic decision aid.

[1]  Elizabeth L. Barry,et al.  Evaluation of a Deep Neural Network for Automated Classification of Colorectal Polyps on Histopathologic Slides , 2020, JAMA network open.

[2]  A. Sanyal,et al.  Comparison of noninvasive markers of fibrosis in patients with nonalcoholic fatty liver disease. , 2009, Clinical gastroenterology and hepatology : the official clinical practice journal of the American Gastroenterological Association.

[3]  S. Alavian,et al.  Inter-observer and Intra-observer Agreement in Pathological Evaluation of Non-alcoholic Fatty Liver Disease Suspected Liver Biopsies , 2014, Hepatitis monthly.

[4]  R. C. Macridis.  A review , 1963 .

[5]  S. Raab.  The cost-effectiveness of immunohistochemistry. , 2000, Archives of pathology & laboratory medicine.

[6]  Peter Bankhead,et al.  QuPath: Open source software for digital pathology image analysis , 2017, Scientific Reports.

[7]  Joshua J. Levy,et al.  Topological Feature Extraction and Visualization of Whole Slide Images using Graph Neural Networks , 2020 .

[8]  Paul J. Barr,et al.  Estimation of an inter-rater intra-class correlation coefficient that overcomes common assumption violations in the assessment of health measurement scales , 2018, BMC Medical Research Methodology.

[9]  C. Wells,et al.  Coding ordinal independent variables in multiple regression analyses. , 1987, American journal of epidemiology.

[10]  Peter Bankhead,et al.  QuPath: Open source software for digital pathology image analysis , 2017 .

[11]  Janne Heikkilä,et al.  Towards Virtual H&E Staining of Hyperspectral Lung Histology Images Using Conditional Generative Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[12]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Jens Rittscher,et al.  The use of digital pathology and image analysis in clinical trials , 2019, The journal of pathology. Clinical research.

[14]  Dorit Merhof,et al.  Supervised and Unsupervised Cell-Nuclei Detection in Immunohistology , 2019 .

[15]  Ulas Bagci,et al.  How to fool radiologists with generative adversarial networks? A visual turing test for lung cancer diagnosis , 2017, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[16]  A. Fischer,et al.  Hematoxylin and eosin staining of tissue and cell sections. , 2008, CSH protocols.

[17]  D. Molavi.  The Practice of Surgical Pathology: A Beginner's Guide to the Diagnostic Process , 2008 .

[18]  Richard McElreath,et al.  Statistical Rethinking: A Bayesian Course with Examples in R and Stan , 2015 .

[19]  Richard J. Chen,et al.  Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images , 2018, IEEE Transactions on Medical Imaging.

[20]  Pabitra Mitra,et al.  Generative Adversarial Learning for Reducing Manual Annotation in Semantic Segmentation on Large Scale Miscroscopy Images: Automated Vessel Segmentation in Retinal Fundus Image as Test Case , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  G. Birk,et al.  Deep learning enables pathologist-like scoring of NASH models , 2019, Scientific Reports.

[22]  M. J. Kendrick,et al.  Diagnostic Accuracy of Virtual Pathology vs Traditional Microscopy in a Large Dermatopathology Study , 2017, JAMA dermatology.

[23]  A. Ozcan,et al.  Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning , 2018, Nature Biomedical Engineering.

[24]  M. Manns,et al.  A comparison of fibrosis progression in chronic liver diseases. , 2003, Journal of hepatology.

[25]  L. Henry,et al.  Mortality Related to Nonalcoholic Fatty Liver Disease Is Increasing in the United States , 2019, Hepatology communications.

[26]  Saeed Hassanpour,et al.  Generative Image Translation for Data Augmentation in Colorectal Histopathology Images , 2019, ML4H@NeurIPS.

[27]  Z. Younossi,et al.  Nonalcoholic fatty liver disease: a spectrum of clinical and pathological severity. , 1999, Gastroenterology.

[28]  L. Ferrell.  Liver Pathology: Cirrhosis, Hepatitis, and Primary Liver Tumors. Update and Diagnostic Problems , 2000, Modern Pathology.

[29]  S. Pol,et al.  FIB‐4: An inexpensive and accurate marker of fibrosis in HCV infection. comparison with liver biopsy and fibrotest , 2007, Hepatology.

[30]  Lassi Paavolainen,et al.  A deep learning framework for nucleus segmentation using image style transfer , 2019, bioRxiv.

[31]  Qianni Zhang,et al.  GAN-based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis , 2019, ArXiv.

[32]  Clive R. Taylor,et al.  Whole Slide Imaging Versus Microscopy for Primary Diagnosis in Surgical Pathology , 2017, The American journal of surgical pathology.

[33]  Christian Robert,et al.  Statistical Rethinking , 2017 .

[34]  April Khademi,et al.  Assessing the Impact of Color Normalization in Convolutional Neural Network-Based Nuclei Segmentation Frameworks , 2019, Front. Bioeng. Biotechnol..

[35]  Saeed Hassanpour,et al.  Finding a Needle in the Haystack: Attention-Based Classification of High Resolution Microscopy Images , 2018, ArXiv.

[36]  A. Pollard,et al.  Limb proportions show developmental plasticity in response to embryo movement , 2017, Scientific Reports.

[37]  C. Byrne,et al.  Complications, morbidity and mortality of nonalcoholic fatty liver disease. , 2020, Metabolism: clinical and experimental.

[38]  Joshua J. Levy,et al.  Topological Feature Extraction and Visualization of Whole Slide Images using Graph Neural Networks , 2020, bioRxiv.

[39]  T. Therneau,et al.  Hyponatremia and mortality among patients on the liver-transplant waiting list. , 2008, The New England journal of medicine.

[40]  Giancarlo Mauri,et al.  GAN-based synthetic brain MR image generation , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[41]  Linda A Bradley,et al.  Principles of analytic validation of immunohistochemical assays: Guideline from the College of American Pathologists Pathology and Laboratory Quality Center. , 2014, Archives of pathology & laboratory medicine.

[42]  K. Batts,et al.  Independent predictors of liver fibrosis in patients with nonalcoholic steatohepatitis , 1999, Hepatology.

[43]  Michael R. Crusoe,et al.  Common Workflow Language , 2015 .

[44]  R. Hultcrantz,et al.  Fibrosis stage but not NASH predicts mortality and time to development of severe liver disease in biopsy-proven NAFLD. , 2017, Journal of hepatology.

[45]  M. Stepanova,et al.  Changes in the prevalence of the most common causes of chronic liver diseases in the United States from 1988 to 2008. , 2011, Clinical gastroenterology and hepatology : the official clinical practice journal of the American Gastroenterological Association.

[46]  Ghassan Hamarneh,et al.  Adversarial Stain Transfer for Histopathology Image Analysis , 2018, IEEE Transactions on Medical Imaging.

[47]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[48]  S. Msika,et al.  Prevalence and Predictive Factors of Non-Alcoholic Steatohepatitis (NASH) in Morbidly Obese Patients Undergoing Bariatric Surgery , 2006, Obesity surgery.

[49]  Alexis B. Carter,et al.  Validating whole slide imaging for diagnostic purposes in pathology: guideline from the College of American Pathologists Pathology and Laboratory Quality Center. , 2013, Archives of pathology & laboratory medicine.

[50]  P. Bürkner,et al.  Modelling monotonic effects of ordinal predictors in Bayesian regression models. , 2020, The British journal of mathematical and statistical psychology.

[51]  Minghua Deng,et al.  A Novel Noninvasive Program for Staging Liver Fibrosis in Untreated Patients With Chronic Hepatitis B , 2019, Clinical and translational gastroenterology.

[52]  J. Montaner,et al.  Development of a simple noninvasive index to predict significant fibrosis in patients with HIV/HCV coinfection , 2006, Hepatology.

[53]  D. Kleiner,et al.  Histology of Nonalcoholic Fatty Liver Disease and Nonalcoholic Steatohepatitis in Adults and Children. , 2016, Clinics in liver disease.

[54]  Yuan Liu,et al.  DermGAN: Synthetic Generation of Clinical Skin Images with Pathology , 2019, ML4H@NeurIPS.

[55]  A. Madabhushi,et al.  Histopathological Image Analysis: A Review , 2009, IEEE Reviews in Biomedical Engineering.

[56]  Anne E Carpenter,et al.  Opportunities and obstacles for deep learning in biology and medicine , 2017, bioRxiv.

[57]  D. Latinne,et al.  Progressive Fibrosis Is Driven by Genetic Predisposition, Allo-immunity, and Inflammation in Pediatric Liver Transplant Recipients , 2016, EBioMedicine.

[58]  Roderick Murray-Smith,et al.  Pathology GAN: Learning deep representations of cancer tissue , 2019, MIDL.

[59]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[60]  R. Wiesner,et al.  The new liver allocation system: Moving toward evidence‐based transplantation policy , 2002, Liver transplantation : official publication of the American Association for the Study of Liver Diseases and the International Liver Transplantation Society.

[61]  Joshua J. Levy,et al.  PathFlowAI: A High-Throughput Workflow for Preprocessing, Deep Learning and Interpretation in Digital Pathology , 2019, PSB.

[62]  Daisuke Komura,et al.  Machine Learning Methods for Histopathological Image Analysis , 2017, Computational and structural biotechnology journal.

[63]  Carl Boettiger,et al.  An introduction to Docker for reproducible research , 2014, OPSR.

[64]  G. Elwyn,et al.  The psychometric properties of Observer OPTION(5), an observer measure of shared decision making. , 2015, Patient education and counseling.

[65]  Joshua J. Levy,et al.  PathFlow-MixMatch for Whole Slide Image Registration: An Investigation of a Segment-Based Scalable Image Registration Method , 2020 .

[66]  L. Adams,et al.  Lifestyle modification in NAFLD/NASH: Facts and figures , 2019, JHEP reports.

[67]  C. Hellerbrand,et al.  Non-alcoholic fatty liver disease, obesity and the metabolic syndrome. , 2014, Best practice & research. Clinical gastroenterology.

[68]  J. S. Long,et al.  Regression models for categorical dependent variables using Stata, 2nd Edition , 2005 .

[69]  A. Berchtold.  Test–retest: Agreement or reliability? , 2016 .

[70]  Navid Borhani,et al.  Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy. , 2019, Biomedical optics express.

[71]  Yang Lei,et al.  MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks. , 2019, Medical physics.

[72]  B. Bell,et al.  A comparison of hepatopathologists' and community pathologists' review of liver biopsy specimens from patients with hepatitis C. , 2009, Clinical gastroenterology and hepatology : the official clinical practice journal of the American Gastroenterological Association.

[73]  Yibo Zhang,et al.  PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning , 2018, Light: Science & Applications.

[74]  Aman Rana,et al.  Computational Histological Staining and Destaining of Prostate Core Biopsy RGB Images with Generative Adversarial Neural Networks , 2018, 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA).

[75]  Peter H. N. de With,et al.  Stain normalization of histopathology images using generative adversarial networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[76]  S. Al-Busafi,et al.  Clinical Manifestations of Portal Hypertension , 2012, International journal of hepatology.

[77]  O. Cummings,et al.  Generalizability of the Nonalcoholic Steatohepatitis Clinical Research Network Histologic Scoring System for Nonalcoholic Fatty Liver Disease , 2011, Journal of clinical gastroenterology.

[78]  Z. Goodman.  Grading and staging systems for inflammation and fibrosis in chronic liver diseases. , 2007, Journal of hepatology.

[79]  Christine Silberhorn,et al.  Metasurface interferometry toward quantum sensors , 2019, Light, science & applications.

[80]  I. Parker.  Facts and figures. , 1973, The Probe.

[81]  A. Barritt,et al.  NAFLD and liver transplantation: Current burden and expected challenges. , 2016, Journal of hepatology.

[82]  R. Karim,et al.  NASH and insulin resistance: Insulin hypersecretion and specific association with the insulin resistance syndrome , 2002, Hepatology.

[83]  Saeed Hassanpour,et al.  Pathologist-level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks , 2019, Scientific Reports.

[84]  M. Delgado-Rodríguez,et al.  Systematic review and meta-analysis. , 2017, Medicina intensiva.

[85]  Christopher R. Jackson,et al.  A machine learning algorithm for simulating immunohistochemistry: development of SOX10 virtual IHC and evaluation on primarily melanocytic neoplasms , 2020, Modern Pathology.

[86]  N. Rajpoot,et al.  Diagnostic concordance and discordance in digital pathology: a systematic review and meta-analysis , 2020, Journal of Clinical Pathology.

[87]  Christopher R. Jackson,et al.  Preliminary Evaluation of the Utility of Deep Generative Histopathology Image Translation at a Mid-Sized NCI Cancer Center , 2020, bioRxiv.

[88]  John Chilton,et al.  Common Workflow Language, v1.0 , 2016 .

[89]  Paul Babyn,et al.  Generative Adversarial Network in Medical Imaging: A Review , 2018, Medical Image Anal..

[90]  Jen‐pei Liu,et al.  Noninferiority Tests Based on Concordance Correlation Coefficient for Assessment of the Agreement for Gene Expression Data from Microarray Experiments , 2007, Journal of biopharmaceutical statistics.

[91]  O. Cummings,et al.  Design and validation of a histological scoring system for nonalcoholic fatty liver disease , 2005, Hepatology.