Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images

Nuclear segmentation and classification within Haematoxylin & Eosin stained histology images is a fundamental prerequisite in the digital pathology work-flow. The development of automated methods for nuclear segmentation and classification enables the quantitative analysis of tens of thousands of nuclei within a whole-slide pathology image, opening up possibilities of further analysis of large-scale nuclear morphometry. However, automated nuclear segmentation and classification is faced with a major challenge in that there are several different types of nuclei, some of them exhibiting large intra-class variability such as the nuclei of tumour cells. Additionally, some of the nuclei are often clustered together. To address these challenges, we present a novel convolutional neural network for simultaneous nuclear segmentation and classification that leverages the instance-rich information encoded within the vertical and horizontal distances of nuclear pixels to their centres of mass. These distances are then utilised to separate clustered nuclei, resulting in an accurate segmentation, particularly in areas with overlapping instances. Then, for each segmented instance the network predicts the type of nucleus via a devoted up-sampling branch. We demonstrate state-of-the-art performance compared to other methods on multiple independent multi-tissue histology image datasets. As part of this work, we introduce a new dataset of Haematoxylin & Eosin stained colorectal adenocarcinoma image tiles, containing 24,319 exhaustively annotated nuclei with associated class labels.

[1]  Olaf Hellwich,et al.  A Multi-resolution Approach for Combining Visual Information using Nuclei Segmentation and Classification in Histopathological Images , 2015, VISAPP.

[2]  Philip H. S. Torr,et al.  On the Robustness of Semantic Segmentation Models to Adversarial Attacks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Anant Madabhushi,et al.  An Integrated Region-, Boundary-, Shape-Based Active Contour for Multiple Object Overlap Resolution in Histological Imagery , 2012, IEEE Transactions on Medical Imaging.

[4]  Xiaobo Zhou,et al.  Nuclei Segmentation Using Marker-Controlled Watershed, Tracking Using Mean-Shift, and Kalman Filter in Time-Lapse Microscopy , 2006, IEEE Transactions on Circuits and Systems I: Regular Papers.

[5]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[6]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[7]  Javier DeFelipe,et al.  Segmentation of neuronal nuclei based on clump splitting and a two-step binarization of images , 2013, Expert Syst. Appl..

[8]  Surabhi Bhargava,et al.  A Dataset and a Technique for Generalized Nuclear Segmentation for Computational Pathology , 2017, IEEE Transactions on Medical Imaging.

[9]  Anil K. Jain,et al.  Prostate cancer detection: Fusion of cytological and textural features , 2011, Journal of pathology informatics.

[10]  Carsten Rother,et al.  Panoptic Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  David B. A. Epstein,et al.  Micro‐Net: A unified model for segmentation of various objects in microscopy images , 2018, Medical Image Anal..

[12]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Nasir M. Rajpoot,et al.  Locality Sensitive Deep Learning for Detection and Classification of Nuclei in Routine Colon Cancer Histology Images , 2016, IEEE Trans. Medical Imaging.

[14]  F. Markowetz,et al.  Quantitative Image Analysis of Cellular Heterogeneity in Breast Tumors Complements Genomic Profiling , 2012, Science Translational Medicine.

[15]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[16]  Christoph Meinel,et al.  Deep Learning for Medical Image Analysis , 2018, Journal of Pathology Informatics.

[17]  Thomas Walter,et al.  Segmentation of Nuclei in Histopathology Images by Deep Regression of the Distance Map , 2019, IEEE Transactions on Medical Imaging.

[18]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[19]  Jin Tae Kwak,et al.  Nucleus detection using gradient orientation information and linear least squares regression , 2015, Medical Imaging.

[20]  Daniel Heim,et al.  Detection and Segmentation of Cell Nuclei in Virtual Microscopy Images: A Minimum-Model Approach , 2012, Scientific Reports.

[21]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[22]  A. Madabhushi,et al.  Nuclear shape and orientation features from H&E images predict survival in early-stage estrogen receptor-positive breast cancers , 2018, Laboratory Investigation.

[23]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[24]  Joel H. Saltz,et al.  Methods for Segmentation and Classification of Digital Microscopy Tissue Images , 2018, Front. Bioeng. Biotechnol..

[25]  Jan Lammerding,et al.  Nuclear mechanics in disease. , 2011, Annual review of biomedical engineering.

[26]  George Lee,et al.  Image analysis and machine learning in digital pathology: Challenges and opportunities , 2016, Medical Image Anal..

[27]  Miao Liao,et al.  Automatic segmentation for cell images based on bottleneck detection and ellipse fitting , 2016, Neurocomputing.

[28]  A. Huisman,et al.  Automatic Nuclei Segmentation in H&E Stained Breast Cancer Histopathology Images , 2013, PloS one.

[29]  Anne E Carpenter,et al.  CellProfiler: image analysis software for identifying and quantifying cell phenotypes , 2006, Genome Biology.

[30]  Thomas Walter,et al.  Nuclei segmentation in histopathology images using deep neural networks , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[31]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  David B. A. Epstein,et al.  Cellular Community Detection for Tissue Phenotyping in Histology Images , 2018, COMPAY/OMIA@MICCAI.

[33]  J. Elmore,et al.  Diagnostic concordance among pathologists interpreting breast biopsy specimens. , 2015, JAMA.

[34]  Pheng-Ann Heng,et al.  CIA-Net: Robust Nuclei Instance Segmentation with Contour-aware Information Aggregation , 2019, IPMI.

[35]  Jagath C. Rajapakse,et al.  Segmentation of Clustered Nuclei With Shape Markers and Marking Function , 2009, IEEE Transactions on Biomedical Engineering.

[36]  Peter Bankhead,et al.  QuPath: Open source software for digital pathology image analysis , 2017, Scientific Reports.

[37]  David B. A. Epstein,et al.  Novel digital signatures of tissue phenotypes for predicting distant metastasis in colorectal cancer , 2018, Scientific Reports.

[38]  A. Madabhushi,et al.  Histopathological Image Analysis: A Review , 2009, IEEE Reviews in Biomedical Engineering.

[39]  Yuxin Cui,et al.  A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images , 2018, Medical & Biological Engineering & Computing.

[40]  Yongming Li,et al.  Automatic cell nuclei segmentation and classification of breast cancer histopathology images , 2016, Signal Process..

[41]  Nasir M. Rajpoot,et al.  A bottom-up approach for tumour differentiation in whole slide images of lung adenocarcinoma , 2018, Medical Imaging.

[42]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Nasir M. Rajpoot,et al.  SAMS-NET: Stain-aware multi-scale network for instance-based nuclei segmentation in histology images , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[44]  Hao Chen,et al.  DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Bahram Parvin,et al.  Deep Learning Models Delineates Multiple Nuclear Phenotypes in H&E Stained Histology Sections , 2018, ArXiv.

[46]  A. Madabhushi,et al.  Spatial Architecture and Arrangement of Tumor-Infiltrating Lymphocytes for Predicting Likelihood of Recurrence in Early-Stage Non–Small Cell Lung Cancer , 2018, Clinical Cancer Research.

[47]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[49]  N. Rajpoot,et al.  Locality Sensitive Deep Learning for Detection and Classification of Nuclei in Routine Colon Cancer Histology Images , 2016, IEEE Transactions on Medical Imaging.