A hybrid network for automatic hepatocellular carcinoma segmentation in H&E-stained whole slide images

Hepatocellular carcinoma (HCC), as the most common type of primary malignant liver cancer, has become a leading cause of cancer deaths in recent years. Accurate segmentation of HCC lesions is critical for tumor load assessment, surgery planning, and postoperative examination. As the appearance of HCC lesions varies greatly across patients, traditional manual segmentation is a very tedious and time-consuming process, the accuracy of which is also difficult to ensure. Therefore, a fully automated and reliable HCC segmentation system is in high demand. In this work, we present a novel hybrid neural network based on multi-task learning and ensemble learning techniques for accurate HCC segmentation of hematoxylin and eosin (H&E)-stained whole slide images (WSIs). First, three task-specific branches are integrated to enlarge the feature space, based on which the network is able to learn more general features and thus reduce the risk of overfitting. Second, an ensemble learning scheme is leveraged to perform feature aggregation, in which selective kernel modules (SKMs) and spatial and channel-wise squeeze-and-excitation modules (scSEMs) are adopted for capturing the features from different spaces and scales. Our proposed method achieves state-of-the-art performance on three publicly available datasets, with segmentation accuracies of 0.797, 0.923, and 0.765 in the PAIP, CRAG, and UHCMC&CWRU datasets, respectively, which demonstrates its effectiveness in addressing the HCC segmentation problem. To the best of our knowledge, this is also the first work on the pixel-wise HCC segmentation of H&E-stained WSIs.

[1]  Ying-Zong Juang,et al.  Automatic HCC Detection Using Convolutional Network with Multi-Magnification Input Images , 2019, 2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS).

[2]  Jian Yang,et al.  Selective Kernel Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Rui Chai,et al.  Large-scale tissue histopathology image segmentation based on feature pyramid , 2018, EURASIP J. Image Video Process..

[4]  Shadi Albarqouni,et al.  AggNet : Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images , 2016 .

[5]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[6]  Tatsuya Harada,et al.  Multi-Stage Pathological Image Classification Using Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[7]  David B. A. Epstein,et al.  Glandular Morphometrics for Objective Grading of Colorectal Adenocarcinoma Histology Images , 2017, Scientific Reports.

[8]  J. D. Wegner,et al.  SEMANTIC SEGMENTATION OF AERIAL IMAGES WITH AN ENSEMBLE OF CNNS , 2016, ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences.

[9]  Hong Liu,et al.  A Fast and Refined Cancer Regions Segmentation Framework in Whole-slide Breast Pathological Images , 2019, Scientific Reports.

[10]  Huchuan Lu,et al.  Enhancing Diversity of Defocus Blur Detectors via Cross-Ensemble Network , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Nima Tajbakhsh,et al.  UNet++: A Nested U-Net Architecture for Medical Image Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[12]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[13]  Saeed Hassanpour,et al.  Deep Learning for Classification of Colorectal Polyps on Whole-slide Images , 2017, Journal of pathology informatics.

[14]  Thomas J. Fuchs,et al.  Clinical-grade computational pathology using weakly supervised deep learning on whole slide images , 2019, Nature Medicine.

[15]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[16]  Max Q.-H. Meng,et al.  A Novel OCR-RCNN for Elevator Button Recognition , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[17]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Cheng Chen,et al.  Selective Feature Aggregation Network with Area-Boundary Constraints for Polyp Segmentation , 2019, MICCAI.

[19]  Linda G. Shapiro,et al.  Learning to Segment Breast Biopsy Whole Slide Images , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[20]  Mrinal K. Mandal,et al.  Automated analysis and diagnosis of skin melanoma on whole slide histopathological images , 2015, Pattern Recognit..

[21]  Hao Chen,et al.  MILD‐Net: Minimal information loss dilated network for gland instance segmentation in colon histology images , 2018, Medical Image Anal..

[22]  Jundong Liu,et al.  Hippocampus segmentation through multi-view ensemble ConvNets , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[23]  L. Terracciano,et al.  Histopathology of hepatocellular carcinoma. , 2014, World journal of gastroenterology.

[24]  Nassir Navab,et al.  Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks , 2018, MICCAI.

[25]  Vahid Khanagha,et al.  Context Aware Lung Cancer Annotation in Whole Slide Images Using Fully Convolutional Neural Networks , 2019, ICIAR.

[26]  Fuyong Xing,et al.  Deep Learning in Microscopy Image Analysis: A Survey. , 2018, IEEE transactions on neural networks and learning systems.

[27]  Shu Liu,et al.  Path Aggregation Network for Instance Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Linda G. Shapiro,et al.  Y-Net: Joint Segmentation and Classification for Diagnosis of Breast Biopsy Images , 2018, MICCAI.

[29]  Wei-Shi Zheng,et al.  Fully convolutional network ensembles for white matter hyperintensities segmentation in MR images , 2018, NeuroImage.

[30]  Angel Cruz-Roa,et al.  High-throughput adaptive sampling for whole-slide histopathology image analysis (HASHI) via convolutional neural networks: Application to invasive breast cancer detection , 2018, PloS one.

[31]  Ling Shao,et al.  Collaborative Learning of Semi-Supervised Segmentation and Classification for Medical Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Hayit Greenspan,et al.  Chest pathology detection using deep learning with non-medical training , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[33]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Malgorzata Lorent,et al.  Deep learning for damaged tissue detection and segmentation in Ki-67 brain tumor specimens based on the U-net model , 2018 .

[35]  Daisuke Komura,et al.  Machine Learning Methods for Histopathological Image Analysis , 2017, Computational and structural biotechnology journal.

[36]  Raghu Machiraju,et al.  Imitating Pathologist Based Assessment With Interpretable and Context Based Neural Network Modeling of Histology Images , 2018, Biomedical informatics insights.

[37]  Tomoharu Kiyuna,et al.  Enhancing automatic classification of hepatocellular carcinoma images through image masking, tissue changes and trabecular features , 2015, Journal of pathology informatics.

[38]  N. Razavian,et al.  Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning , 2018, Nature Medicine.

[39]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[40]  Geert J. S. Litjens,et al.  Automatic segmentation of histopathological slides of renal tissue using deep learning , 2018, Medical Imaging.

[41]  G. Wainrib,et al.  Deep learning-based classification of mesothelioma improves prediction of patient outcome , 2019, Nature Medicine.

[42]  Hong Liu,et al.  WSI-Net: Branch-Based and Hierarchy-Aware Network for Segmentation and Classification of Breast Histopathological Whole-Slide Images , 2019, MLMI@MICCAI.

[43]  Jie Zhao,et al.  Automated Segmentation Of Cervical Nuclei In Pap Smear Images Using Deformable Multi-Path Ensemble Model , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[44]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[45]  Ryoma Bise,et al.  Adaptive Weighting Multi-Field-Of-View CNN for Semantic Segmentation in Pathology , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Peng Tang,et al.  Multi-proportion channel ensemble model for retinal vessel segmentation , 2019, Comput. Biol. Medicine.

[47]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[48]  Chunhua Shen,et al.  Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Ling Shao,et al.  DME-Net: Diabetic Macular Edema Grading by Auxiliary Task Learning , 2019, MICCAI.

[50]  Sijia Wang,et al.  Classification and Segmentation of Hyperspectral Data of Hepatocellular Carcinoma Samples Using 1‐D Convolutional Neural Network , 2019, Cytometry. Part A : the journal of the International Society for Analytical Cytology.

[51]  Linda G. Shapiro,et al.  ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation , 2018, ECCV.

[52]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Alain Pitiot,et al.  A cascade-learning approach for automated segmentation of tumour epithelium in colorectal cancer , 2019, Expert Syst. Appl..

[54]  Zhuowen Tu,et al.  Weakly supervised histopathology cancer image segmentation and classification , 2014, Medical Image Anal..

[55]  Jakob Nikolas Kather,et al.  Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer , 2019, Nature Medicine.

[56]  Arkadiusz Gertych,et al.  Semantic segmentation for prostate cancer grading by convolutional neural networks , 2018, Medical Imaging.

[57]  Konstantinos Kamnitsas,et al.  Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation , 2017, BrainLes@MICCAI.

[58]  David B. A. Epstein,et al.  Tumor Segmentation in Whole Slide Images Using Persistent Homology and Deep Convolutional Features , 2017, MIUA.

[59]  Michael Kampffmeyer,et al.  Reinforced Auto-Zoom Net: Towards Accurate and Fast Breast Cancer Segmentation in Whole-slide Images , 2018, DLMIA/ML-CDS@MICCAI.

[60]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[61]  Egor Krivov,et al.  Ensembling Neural Networks for Digital Pathology Images Classification and Segmentation , 2018, ICIAR.

[62]  David B. A. Epstein,et al.  Fast and Accurate Tumor Segmentation of Histology Images using Persistent Homology and Deep Convolutional Features , 2018, Medical Image Anal..

[63]  Kay R. J. Oskal,et al.  A U-net based approach to epidermal tissue segmentation in whole slide histopathological images , 2019, SN Applied Sciences.