Computational tissue staining of non-linear multimodal imaging using supervised and unsupervised deep learning

Hematoxylin and Eosin (H&E) staining is the ’gold-standard’ method in histopathology. However, standard H&E staining of high-quality tissue sections requires long sample preparation times including sample embedding, which restricts its application for ’real-time’ disease diagnosis. Due to this reason, a label-free alternative technique like non-linear multimodal (NLM) imaging, which is the combination of three non-linear optical modalities including coherent anti-Stokes Raman scattering, two-photon excitation fluorescence and second-harmonic generation, is proposed in this work. To correlate the information of the NLM images with H&E images, this work proposes computational staining of NLM images using deep learning models in a supervised and an unsupervised approach. In the supervised and the unsupervised approach, conditional generative adversarial networks (CGANs) and cycle conditional generative adversarial networks (cycle CGANs) are used, respectively. Both CGAN and cycle CGAN models generate pseudo H&E images, which are quantitatively analyzed based on mean squared error, structure similarity index and color shading similarity index. The mean of the three metrics calculated for the computationally generated H&E images indicate significant performance. Thus, utilizing CGAN and cycle CGAN models for computational staining is beneficial for diagnostic applications without performing a laboratory-based staining procedure. To the author’s best knowledge, it is the first time that NLM images are computationally stained to H&E images using GANs in an unsupervised manner.

[1]  Jürgen Popp,et al.  Multimodal Imaging Spectroscopy of Tissue. , 2015, Annual review of analytical chemistry.

[2]  Janne Heikkilä,et al.  Towards Virtual H&E Staining of Hyperspectral Lung Histology Images Using Conditional Generative Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[3]  B Dietzek,et al.  Multimodal mapping of human skin , 2013, The British journal of dermatology.

[4]  A. Ozcan,et al.  Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning , 2018, Nature Biomedical Engineering.

[5]  Riccardo Cicchi,et al.  Multimodal nonlinear microscopy: A powerful label-free method for supporting standard diagnostics on biological tissues , 2014 .

[6]  Nicholas Ayache,et al.  The Correlation Ratio as a New Similarity Measure for Multimodal Image Registration , 1998, MICCAI.

[7]  Lin Shi,et al.  Color quantification for evaluation of stained tissues , 2011, Cytometry. Part A : the journal of the International Society for Analytical Cytology.

[8]  Francesco S. Pavone,et al.  Multimodal image analysis in tissue diagnostics for skin melanoma , 2018 .

[9]  Juho Kannala,et al.  Deep learning for magnification independent breast cancer histopathology image classification , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[10]  Tejas Sudharshan Mathai,et al.  Accurate Tissue Interface Segmentation via Adversarial Pre-Segmentation of Anterior Segment OCT Images , 2019, Biomedical optics express.

[11]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[12]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[13]  Michael Schmitt,et al.  Semantic Segmentation of Non-linear Multimodal Images for Disease Grading of Inflammatory Bowel Disease: A SegNet-based Application , 2019, ICPRAM.

[14]  Jürgen Popp,et al.  Nonlinear microscopy, infrared, and Raman microspectroscopy for brain tumor analysis. , 2011, Journal of biomedical optics.

[15]  Jürgen Popp,et al.  Beyond endoscopic assessment in inflammatory bowel disease: real-time histology of disease activity by non-linear multimodal imaging , 2016, Scientific Reports.

[16]  A. Ozcan,et al.  Cross-Modality Deep Learning Achieves Super-Resolution in Fluorescence Microscopy , 2019, 2019 Conference on Lasers and Electro-Optics (CLEO).

[17]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[18]  Yibo Zhang,et al.  Deep learning‐based color holographic microscopy , 2019, Journal of biophotonics.

[19]  Zhou Wang,et al.  Translation insensitive image similarity in complex wavelet domain , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[20]  Todd C. Hollon,et al.  Rapid intraoperative histology of unprocessed surgical specimens via fibre-laser-based stimulated Raman scattering microscopy , 2017, Nature Biomedical Engineering.

[21]  Jürgen Popp,et al.  Pseudo-HE images derived from CARS/TPEF/SHG multimodal imaging in combination with Raman-spectroscopy as a pathological screening tool , 2016, BMC Cancer.

[22]  Jia Li,et al.  Automatic Sketch Colorization with Tandem Conditional Adversarial Networks , 2018, 2018 11th International Symposium on Computational Intelligence and Design (ISCID).

[23]  Xinjian Chen,et al.  Speckle noise reduction in optical coherence tomography images based on edge-sensitive cGAN. , 2018, Biomedical optics express.

[24]  Helmut Neumann,et al.  Label-free imaging of inflammatory bowel disease using multiphoton microscopy. , 2013, Gastroenterology.

[25]  Bahram Parvin,et al.  Automated Histology Analysis: Opportunities for signal processing , 2015, IEEE Signal Processing Magazine.

[26]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[27]  Su Ruan,et al.  Medical Image Synthesis with Context-Aware Generative Adversarial Networks , 2016, MICCAI.

[28]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Hiroshi Ishikawa,et al.  Retinal optical coherence tomography image enhancement via deep learning. , 2018, Biomedical optics express.

[30]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[31]  Ian J. Goodfellow,et al.  NIPS 2016 Tutorial: Generative Adversarial Networks , 2016, ArXiv.

[32]  Xun Xu,et al.  Improved cGAN based linear lesion segmentation in high myopia ICGA images. , 2019, Biomedical optics express.

[33]  B. Dietzek,et al.  Detection and Discrimination of Non-Melanoma Skin Cancer by Multimodal Imaging , 2013, Healthcare.

[34]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[35]  Tomer Michaeli,et al.  Deep-STORM: super-resolution single-molecule microscopy by deep learning , 2018, 1801.09631.

[36]  Jürgen Popp,et al.  Multimodal nonlinear microscopy of head and neck carcinoma — toward surgery assisting frozen section analysis , 2016, Head & neck.

[37]  Ghassan Hamarneh,et al.  Adversarial Stain Transfer for Histopathology Image Analysis , 2018, IEEE Transactions on Medical Imaging.

[38]  Peng Fei,et al.  High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network. , 2019, Biomedical optics express.

[39]  Qianni Zhang,et al.  GAN-based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis , 2019, ArXiv.

[40]  Jürgen Popp,et al.  Multimodal nonlinear microscopic investigations on head and neck squamous cell carcinoma: Toward intraoperative imaging , 2013, Head & neck.

[41]  Bin Li,et al.  Detection of exudates in fundus photographs with imbalanced learning using conditional generative adversarial network. , 2018, Biomedical optics express.

[42]  Hyunseok Min,et al.  Neural Stain-Style Transfer Learning using GAN for Histopathological Images , 2017, ArXiv.

[43]  Jie Tian,et al.  Improved generative adversarial networks using the total gradient loss for the resolution enhancement of fluorescence images. , 2019, Biomedical optics express.

[44]  Shuxia Guo,et al.  Deep learning a boon for biophotonics? , 2020, Journal of biophotonics.