Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology

One of the biggest challenges for applying machine learning to histopathology is weak supervision: whole-slide images have billions of pixels yet often only one global label. The state of the art therefore relies on strongly-supervised model training using additional local annotations from domain experts. However, in the absence of detailed annotations, most weakly-supervised approaches depend on a frozen feature extractor pre-trained on ImageNet. We identify this as a key weakness and propose to train an in-domain feature extractor on histology images using MoCo v2, a recent self-supervised learning algorithm. Experimental results on Camelyon16 and TCGA show that the proposed extractor greatly outperforms its ImageNet counterpart. In particular, our results improve the weakly-supervised state of the art on Camelyon16 from 91.4% to 98.7% AUC, thereby closing the gap with strongly-supervised models that reach 99.3% AUC. Through these experiments, we demonstrate that feature extractors trained via self-supervised learning can act as drop-in replacements to significantly improve existing machine learning techniques in histology. Lastly, we show that the learned embedding space exhibits biologically meaningful separation of tissue structures.

[1]  Julien Mairal,et al.  Unsupervised Learning of Visual Features by Contrasting Cluster Assignments , 2020, NeurIPS.

[2]  Konstantinos N. Plataniotis,et al.  HistoSegNet: Semantic Segmentation of Histological Tissue Type in Whole Slide Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[3]  Jakob Nikolas Kather,et al.  Pan-cancer image-based detection of clinically actionable genetic alterations , 2019, Nature Cancer.

[4]  Eric W. Tramel,et al.  Classification and Disease Localization in Histopathology Using Only Global Labels: A Weakly-Supervised Approach , 2018, ArXiv.

[5]  Oriol Vinyals,et al.  Representation Learning with Contrastive Predictive Coding , 2018, ArXiv.

[6]  Dayong Wang,et al.  Deep Learning for Identifying Metastatic Breast Cancer , 2016, ArXiv.

[7]  Gilles Wainrib,et al.  Abstract 2105: HE2RNA: A deep learning model for transcriptomic learning from digital pathology , 2020 .

[8]  Francesco Ciompi,et al.  Neural Image Compression for Gigapixel Histopathology Image Analysis , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Alexander T. Pearson,et al.  Clinical-grade Detection of Microsatellite Instability in Colorectal Tumors by Deep Learning. , 2020, Gastroenterology.

[10]  Kaiming He,et al.  Momentum Contrast for Unsupervised Visual Representation Learning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  K. Sirinukunwattana,et al.  Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning , 2020, Gut.

[12]  Thomas J. Fuchs,et al.  Clinical-grade computational pathology using weakly supervised deep learning on whole slide images , 2019, Nature Medicine.

[13]  Benjamin Bird,et al.  Detection of breast micro-metastases in axillary lymph nodes by infrared micro-spectral imaging. , 2009, The Analyst.

[14]  Andrew H. Beck,et al.  Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer , 2017, JAMA.

[15]  Michal Valko,et al.  Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , 2020, NeurIPS.

[16]  Matthieu Cord,et al.  WELDON: Weakly Supervised Learning of Deep Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Ming Y. Lu,et al.  Semi-supervised breast cancer histology classification using deep multiple instance learning and contrast predictive coding (Conference Presentation) , 2020, Medical Imaging: Digital Pathology.

[18]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[19]  Bram van Ginneken,et al.  Streaming convolutional neural networks for end-to-end learning with multi-megapixel images , 2020, IEEE transactions on pattern analysis and machine intelligence.

[20]  Rajarsi R. Gupta,et al.  Spatial Organization and Molecular Correlation of Tumor-Infiltrating Lymphocytes Using Deep Learning on Pathology Images. , 2018, Cell reports.

[21]  Luca Maria Gambardella,et al.  Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks , 2013, MICCAI.

[22]  Geoffrey E. Hinton,et al.  Big Self-Supervised Models are Strong Semi-Supervised Learners , 2020, NeurIPS.

[23]  B. van Ginneken,et al.  Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study. , 2020, The Lancet. Oncology.

[24]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[26]  Ming Y. Lu,et al.  Semi-Supervised Histology Classification using Deep Multiple Instance Learning and Contrastive Predictive Coding , 2019, ArXiv.

[27]  Max Welling,et al.  Attention-based Deep Multiple Instance Learning , 2018, ICML.

[28]  Kaiming He,et al.  Improved Baselines with Momentum Contrastive Learning , 2020, ArXiv.

[29]  Jitendra Jonnagaddala,et al.  Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks , 2020, Medical Image Anal..

[30]  Jeroen van der Laak,et al.  Detection of Prostate Cancer in Whole-Slide Images Through End-to-End Training With Image-Level Labels , 2020, IEEE Transactions on Medical Imaging.

[31]  Oumeima Laifa,et al.  Predicting Survival After Hepatocellular Carcinoma Resection Using Deep Learning on Histological Slides , 2020, Hepatology.

[32]  Jakob Nikolas Kather,et al.  Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer , 2019, Nature Medicine.

[33]  N. Razavian,et al.  Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning , 2018, Nature Medicine.

[34]  Jeffrey S. Morris,et al.  The Consensus Molecular Subtypes of Colorectal Cancer , 2015, Nature Medicine.