Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks

Cancer prognostication is a challenging task in computational pathology that requires context-aware representations of histology features to adequately infer patient survival. Despite the advancements made in weakly-supervised deep learning, many approaches are not context-aware and are unable to model important morphological feature interactions between cell identities and tissue types that are prognostic for patient survival. In this work, we present Patch-GCN, a contextaware, spatially-resolved patch-based graph convolutional network that hierarchically aggregates instance-level histology features to model localand global-level topological structures in the tumor microenvironment. We validate Patch-GCN with 4,370 gigapixel WSIs across five different cancer types from the Cancer Genome Atlas (TCGA), and demonstrate that Patch-GCN outperforms all prior weakly-supervised approaches by 3.58-9.46%. Our code and corresponding models are publicly available at https://github.com/mahmoodlab/Patch-GCN.

[1]  Rajarsi R. Gupta,et al.  Spatial Organization and Molecular Correlation of Tumor-Infiltrating Lymphocytes Using Deep Learning on Pathology Images. , 2018, Cell reports.

[2]  Andrew H. Beck,et al.  Systematic Analysis of Breast Cancer Morphology Uncovers Stromal Features Associated with Survival , 2011, Science Translational Medicine.

[3]  Ruoyu Li,et al.  Graph CNN for Survival Analysis on Whole Slide Pathological Images , 2018, MICCAI.

[4]  Jean-Philippe Thiran,et al.  Quantifying Explainers of Graph Neural Networks in Computational Pathology , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Ming Y. Lu,et al.  Federated learning for computational pathology on gigapixel whole slide images , 2020, Medical Image Anal..

[6]  Junzhou Huang,et al.  Graph Attention Multi-instance Learning for Accurate Colorectal Cancer Staging , 2020, MICCAI.

[7]  Shaoqun Zeng,et al.  From Detection of Individual Metastases to Classification of Lymph Node Status at the Patient Level: The CAMELYON17 Challenge , 2019, IEEE Transactions on Medical Imaging.

[8]  Fan Yang,et al.  Predicting Lymph Node Metastasis Using Histopathological Images Based on Multiple Instance Learning With Deep Graph Convolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Naoufel Werghi,et al.  Multiplex Cellular Communities in Multi-Gigapixel Colorectal Cancer Histology Images for Tissue Phenotyping , 2020, IEEE Transactions on Image Processing.

[10]  Nasir M. Rajpoot,et al.  A Novel Digital Score for Abundance of Tumour Infiltrating Lymphocytes Predicts Disease Free Survival in Oral Squamous Cell Carcinoma , 2019, Scientific Reports.

[11]  Alexander J. Smola,et al.  Deep Sets , 2017, 1703.06114.

[12]  Anant Madabhushi,et al.  Cell cluster graph for prediction of biochemical recurrence in prostate cancer patients from tissue microarrays , 2013, Medical Imaging.

[13]  Jitendra Jonnagaddala,et al.  Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks , 2020, Medical Image Anal..

[14]  Andrew H. Beck,et al.  Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes , 2021, Nature Communications.

[15]  Junzhou Huang,et al.  WSISA: Making Survival Prediction from Whole Slide Histopathological Images , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Ming Y. Lu,et al.  Data-efficient and weakly supervised computational pathology on whole-slide images , 2020, Nature Biomedical Engineering.

[17]  Ce Zhang,et al.  Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features , 2016, Nature Communications.

[18]  G. Wainrib,et al.  Deep learning-based classification of mesothelioma improves prediction of patient outcome , 2019, Nature Medicine.

[19]  Ming Y. Lu,et al.  AI-based pathology predicts origins for cancers of unknown primary , 2020, Nature.

[20]  Richard J. Chen,et al.  Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images , 2018, IEEE Transactions on Medical Imaging.

[21]  Matthias Schmid,et al.  Bias in Cross-Entropy-Based Training of Deep Survival Networks , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Pheng-Ann Heng,et al.  CGC-Net: Cell Graph Convolutional Network for Grading of Colorectal Cancer Histology Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[23]  Thomas J. Fuchs,et al.  Clinical-grade computational pathology using weakly supervised deep learning on whole slide images , 2019, Nature Medicine.

[24]  Peter Regitnig,et al.  Interpretable survival prediction for colorectal cancer using deep learning , 2021, npj Digital Medicine.

[25]  Nasir Rajpoot,et al.  Capturing Cellular Topology in Multi-Gigapixel Pathology Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[26]  Ming Y. Lu,et al.  Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis , 2019, IEEE Transactions on Medical Imaging.

[27]  T. Hagemann,et al.  The tumor microenvironment at a glance , 2012, Journal of Cell Science.

[28]  Max Welling,et al.  Attention-based Deep Multiple Instance Learning , 2018, ICML.

[29]  Shaoting Zhang,et al.  Feature-Enhanced Graph Networks for Genetic Mutational Prediction Using Histopathological Images in Colon Cancer , 2020, MICCAI.

[30]  Ming Y. Lu,et al.  Semi-Supervised Histology Classification using Deep Multiple Instance Learning and Contrastive Predictive Coding , 2019, ArXiv.

[31]  Ming Y. Lu,et al.  Weakly Supervised Prostate Tma Classification Via Graph Convolutional Networks , 2019, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).

[32]  A. Madabhushi,et al.  Nuclear shape and orientation features from H&E images predict survival in early-stage estrogen receptor-positive breast cancers , 2018, Laboratory Investigation.

[33]  Bernard Ghanem,et al.  DeepGCNs: Can GCNs Go As Deep As CNNs? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[34]  Angela E. Leek,et al.  Geospatial immune variability illuminates differential evolution of lung adenocarcinoma , 2020, Nature Medicine.

[35]  Shrey Gadiya,et al.  Histographs: Graphs in Histopathology , 2019, Medical Imaging: Digital Pathology.

[36]  D. Brat,et al.  Predicting cancer outcomes from histology and genomics using convolutional networks , 2017, Proceedings of the National Academy of Sciences.