HistoCartography: A Toolkit for Graph Analytics in Digital Pathology

Advances in entity-graph analysis of histopathology images have brought in a new paradigm to describe tissue composition, and learn the tissue structure-to-function relationship. Entity-graphs offer flexible and scalable representations to characterize tissue organization, while allowing the incorporation of prior pathological knowledge to further support model explainability. However, their analysis requires prerequisites for image-to-graph translation and knowledge of state-of-the-art algorithms applied to graph-structured data, which can potentially hinder their adoption. In this work, we aim to alleviate these issues by developing HistoCartography, a standardized python API with necessary preprocessing, machine learning and explainability tools to facilitate graph-analytics in computational pathology. Further, we have benchmarked the computational time and performance on multiple datasets across different imaging types and histopathology tasks to highlight the applicability of the API for building computational pathology workflows. HistoCartography is available at https://github.com/histocartography/histocartography.

[1]  Ming Y. Lu,et al.  Data-efficient and weakly supervised computational pathology on whole-slide images , 2020, Nature Biomedical Engineering.

[2]  Deepak Anand,et al.  Visualization for Histopathology Images using Graph Convolutional Neural Networks , 2020, 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE).

[3]  Joshua J. Levy,et al.  Topological Feature Extraction and Visualization of Whole Slide Images using Graph Neural Networks , 2020, bioRxiv.

[4]  Orcun Goksel,et al.  HACT-Net: A Hierarchical Cell-to-Tissue Graph Neural Network for Histopathological Image Classification , 2020, UNSURE/GRAIL@MICCAI.

[5]  J. S. Marron,et al.  A method for normalizing histology slides for quantitative analysis , 2009, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[6]  Ming Y. Lu,et al.  Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis , 2019, IEEE Transactions on Medical Imaging.

[7]  Geert J. S. Litjens,et al.  Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology , 2019, Medical Image Anal..

[8]  Catarina Eloy,et al.  BACH: Grand Challenge on Breast Cancer Histology Images , 2018, Medical Image Anal..

[9]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[10]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Jure Leskovec,et al.  GNNExplainer: Generating Explanations for Graph Neural Networks , 2019, NeurIPS.

[12]  Sébastien Marcel,et al.  Torchvision the machine-vision package of torch , 2010, ACM Multimedia.

[13]  Jean-Philippe Thiran,et al.  Learning Whole-Slide Segmentation from Inexact and Incomplete Labels using Tissue Graphs , 2021, MICCAI.

[14]  Pheng-Ann Heng,et al.  CGC-Net: Cell Graph Convolutional Network for Grading of Colorectal Cancer Histology Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[15]  Cigdem Demir,et al.  The cell graphs of cancer , 2004, ISMB/ECCB.

[16]  Linda Studer,et al.  Classification of Intestinal Gland Cell-Graphs Using Graph Neural Networks , 2021, 2020 25th International Conference on Pattern Recognition (ICPR).

[17]  Thomas J. Fuchs,et al.  Clinical-grade computational pathology using weakly supervised deep learning on whole slide images , 2019, Nature Medicine.

[18]  Joshua J. Levy,et al.  Topological Feature Extraction and Visualization of Whole Slide Images using Graph Neural Networks , 2020 .

[19]  Robert Schwarzenberg,et al.  Layerwise Relevance Visualization in Convolutional Text Graph Classifiers , 2019, EMNLP.

[20]  Pietro Liò,et al.  Principal Neighbourhood Aggregation for Graph Nets , 2020, NeurIPS.

[21]  Xavier Bresson,et al.  Benchmarking Graph Neural Networks , 2020, ArXiv.

[22]  Jean-Philippe Thiran,et al.  Quantifying Explainers of Graph Neural Networks in Computational Pathology , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Orcun Goksel,et al.  Towards Explainable Graph Representations in Digital Pathology , 2020, ArXiv.

[24]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[25]  Nasir M. Rajpoot,et al.  Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer Histology Images , 2019, IEEE Transactions on Medical Imaging.

[26]  Hamid R. Tizhoosh,et al.  Representation Learning of Histopathology Images using Graph Neural Networks , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[27]  Vineeth N. Balasubramanian,et al.  Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[28]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[29]  Ming Y. Lu,et al.  Weakly Supervised Prostate Tma Classification Via Graph Convolutional Networks , 2019, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).

[30]  Francesco Ciompi,et al.  Neural Image Compression for Gigapixel Histopathology Image Analysis , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Selim Aksoy,et al.  Graph convolutional networks for region of interest classification in breast histopathology , 2020, Medical Imaging: Digital Pathology.

[32]  K. Jarrod Millman,et al.  Array programming with NumPy , 2020, Nat..

[33]  Olaf Hellwich,et al.  Cell nuclei attributed relational graphs for efficient representation and classification of gastric cancer in digital histopathology , 2016, SPIE Medical Imaging.

[34]  Fan Yang,et al.  Predicting Lymph Node Metastasis Using Histopathological Images Based on Multiple Instance Learning With Deep Graph Convolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Lars Petersson,et al.  A Survey on Graph-Based Deep Learning for Computational Histopathology , 2021, Comput. Medical Imaging Graph..

[36]  Rafael Molina,et al.  Going deeper through the Gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detection , 2020, Comput. Methods Programs Biomed..

[37]  Alex Smola,et al.  Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs , 2019, ArXiv.

[38]  Ruedi Aebersold,et al.  A curated collection of tissue microarray images and clinical outcome data of prostate cancer patients , 2017, Scientific Data.

[39]  Jean-Philippe Thiran,et al.  Hierarchical graph representations in digital pathology , 2021, Medical Image Anal..

[40]  Nico Karssemeijer,et al.  A multi-scale superpixel classification approach to the detection of regions of interest in whole slide histopathology images , 2015, Medical Imaging.

[41]  Ruoyu Li,et al.  Graph CNN for Survival Analysis on Whole Slide Pathological Images , 2018, MICCAI.

[42]  Anant Madabhushi,et al.  A lymphocyte spatial distribution graph-based method for automated classification of recurrence risk on lung cancer images , 2017, Symposium on Medical Information Processing and Analysis.

[43]  Heiko Hoffmann,et al.  Explainability Methods for Graph Convolutional Neural Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[45]  Olaf Hellwich,et al.  A Comparative Study of Cell Nuclei Attributed Relational Graphs for Knowledge Description and Categorization in Histopathological Gastric Cancer Whole Slide Images , 2017, 2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS).

[46]  Shrey Gadiya,et al.  Histographs: Graphs in Histopathology , 2019, Medical Imaging: Digital Pathology.

[47]  Jin Tae Kwak,et al.  Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images , 2018, Medical Image Anal..

[48]  Nassir Navab,et al.  Structure-Preserving Color Normalization and Sparse Stain Separation for Histological Images , 2016, IEEE Transactions on Medical Imaging.