Seamless Virtual Whole Slide Image Synthesis and Validation Using Perceptual Embedding Consistency

Stain virtualization is an application with growing interest in digital pathology allowing simulation of stained tissue images thus saving lab and tissue resources. Thanks to the success of Generative Adversarial Networks (GANs) and the progress of unsupervised learning, unsupervised style transfer GANs have been successfully used to generate realistic, clinically meaningful and interpretable images. The large size of high resolution Whole Slide Images (WSIs) presents an additional computational challenge. This makes tilewise processing necessary during training and inference of deep learning networks. Instance normalization has a substantial positive effect in style transfer GAN applications but with tilewise inference, it has the tendency to cause a tiling artifact in reconstructed WSIs. In this paper we propose a novel perceptual embedding consistency (PEC) loss forcing the network to learn color, contrast and brightness invariant features in the latent space and hence substantially reducing the aforementioned tiling artifact. Our approach results in more seamless reconstruction of the virtual WSIs. We validate our method quantitatively by comparing the virtually generated images to their corresponding consecutive real stained images.We compare our results to state-of-the-art unsupervised style transfer methods and to the measures obtained from consecutive real stained tissue slide images. We demonstrate our hypothesis about the effect of the PEC loss by comparing model robustness to color, contrast and brightness perturbations and visualizing bottleneck embeddings. We validate the robustness of the bottleneck feature maps by measuring their sensitivity to the different perturbations and using them in a tumor segmentation task. Additionally, we propose a preliminary validation of the virtual staining application by comparing interpretation of 2 pathologists on real and virtual tiles and inter-pathologist agreement.

[1]  Yibo Zhang,et al.  PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning , 2018, Light: Science & Applications.

[2]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Joe W. Gray,et al.  SHIFT: speedy histopathological-to-immunofluorescent translation of whole slide images using conditional generative adversarial networks , 2018, Medical Imaging.

[4]  Nassir Navab,et al.  Virtualization of tissue staining in digital pathology using an unsupervised deep learning approach , 2018, ECDP.

[5]  M. McHugh Interrater reliability: the kappa statistic , 2012, Biochemia medica.

[6]  Nicolas Brieu,et al.  DASGAN - Joint Domain Adaptation and Segmentation for the Analysis of Epithelial Regions in Histopathology PD-L1 Images , 2019, ArXiv.

[7]  S. Umeyama,et al.  Least-Squares Estimation of Transformation Parameters Between Two Point Patterns , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Daniel S Gareau,et al.  Feasibility of digitally stained multimodal confocal mosaics to simulate histopathology. , 2009, Journal of biomedical optics.

[9]  Navid Borhani,et al.  Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy. , 2019, Biomedical optics express.

[10]  Janne Heikkilä,et al.  Towards Virtual H&E Staining of Hyperspectral Lung Histology Images Using Conditional Generative Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[11]  Hidekata Hontani,et al.  Re-staining Pathology Images by FCNN , 2019, 2019 16th International Conference on Machine Vision Applications (MVA).

[12]  K. Gwet Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity , 2002 .

[13]  Rebecca Richards-Kortum,et al.  Confocal fluorescence microscopy for rapid evaluation of invasive tumor cellularity of inflammatory breast carcinoma core needle biopsies , 2014, Breast Cancer Research and Treatment.

[14]  Aman Rana,et al.  Computational Histological Staining and Destaining of Prostate Core Biopsy RGB Images with Generative Adversarial Neural Networks , 2018, 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA).

[15]  Samuel J. Yang,et al.  In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images , 2018, Cell.

[16]  Nassir Navab,et al.  Staingan: Stain Style Transfer for Digital Histological Images , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[17]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[18]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Bruce R. Rosen,et al.  Image reconstruction by domain-transform manifold learning , 2017, Nature.

[20]  A. Madabhushi,et al.  Histopathological Image Analysis: A Review , 2009, IEEE Reviews in Biomedical Engineering.

[21]  Samy Bengio,et al.  Revisiting Distributed Synchronous SGD , 2016, ArXiv.

[22]  Adam Finkelstein,et al.  PairedCycleGAN: Asymmetric Style Transfer for Applying and Removing Makeup , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[23]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[24]  Thomas Kelly,et al.  Fibroblast activation protein-α: a key modulator of the microenvironment in multiple pathologies. , 2012, International review of cell and molecular biology.

[25]  Oliver Grimm,et al.  Enabling Histopathological Annotations on Immunofluorescent Images through Virtualization of Hematoxylin and Eosin , 2018, Journal of pathology informatics.

[26]  Martin J. Yaffe,et al.  Automated multi-class ground-truth labeling of H&E images for deep learning using multiplexed fluorescence microscopy , 2019, Medical Imaging.

[27]  Gregory R. Johnson,et al.  Label-free prediction of three-dimensional fluorescence images from transmitted light microscopy , 2018, Nature Methods.

[28]  Geert J. S. Litjens,et al.  Stain-Transforming Cycle-Consistent Generative Adversarial Networks for Improved Segmentation of Renal Histopathology , 2018, MIDL.

[29]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[30]  M. Rajadhyaksha,et al.  Confocal mosaicing microscopy of human skin ex vivo: spectral analysis for digital staining to simulate histology-like appearance. , 2011, Journal of biomedical optics.

[31]  Michael Gadermayr,et al.  Which Way Round? A Study on the Performance of Stain-Translation for Segmenting Arbitrarily Dyed Histological Images , 2018, MICCAI.

[32]  Inbar Mosseri,et al.  XGAN: Unsupervised Image-to-Image Translation for many-to-many Mappings , 2017, Domain Adaptation for Visual Understanding.

[33]  James G. Fujimoto,et al.  Assessment of breast pathologies using nonlinear microscopy , 2014, Proceedings of the National Academy of Sciences.

[34]  Zhou Wang,et al.  Complex Wavelet Structural Similarity: A New Image Similarity Index , 2009, IEEE Transactions on Image Processing.

[35]  Qianni Zhang,et al.  GAN-based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis , 2019, ArXiv.

[36]  A. Ozcan,et al.  Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning , 2018, Nature Biomedical Engineering.

[37]  Shizuo Kaji,et al.  Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging , 2019, Radiological Physics and Technology.

[38]  Nassir Navab,et al.  Perceptual Embedding Consistency for Seamless Reconstruction of Tilewise Style Transfer , 2019, MICCAI.

[39]  Joachim Hornegger,et al.  Virtual Hematoxylin and Eosin Transillumination Microscopy Using Epi-Fluorescence Imaging , 2016, PloS one.

[40]  Nasir M. Rajpoot,et al.  A Nonlinear Mapping Approach to Stain Normalization in Digital Histopathology Images Using Image-Specific Color Deconvolution , 2014, IEEE Transactions on Biomedical Engineering.