Hybrid guiding: A multi-resolution refinement approach for semantic segmentation of gigapixel histopathological images

Over the past decades, histopathological cancer diagnostics has become more complex, and the increasing number of biopsies is a challenge for most pathology laboratories. Thus, development of automatic methods for evaluation of histopathological cancer sections would be of value. In this study, we used 624 whole slide images (WSIs) of breast cancer from a Norwegian cohort. We propose a cascaded convolutional neural network design, called H2G-Net, for semantic segmentation of gigapixel histopathological images. The design involves a detection stage using a patch-wise method, and a refinement stage using a convolutional autoencoder. To validate the design, we conducted an ablation study to assess the impact of selected components in the pipeline on tumour segmentation. Guiding segmentation, using hierarchical sampling and deep heatmap refinement, proved to be beneficial when segmenting the histopathological images. We found a significant improvement when using a refinement network for postprocessing the generated tumour segmentation heatmaps. The overall best design achieved a Dice score of 0.933± 0.069 on an independent test set of 90 WSIs. The design outperformed single-resolution approaches, such as cluster-guided, patch-wise high-resolution classification using MobileNetV2 (0.872±0.092) and a low-resolution U-Net (0.874±0.128). In addition, the design performed consistently on WSIs across all histological grades and segmentation on a representative ×400 WSI took ∼ 58 seconds, using only the central processing unit. The findings demonstrate the potential of utilizing a refinement network to improve patch-wise predictions. The solution is efficient and does not require overlapping patch inference or ensembling. Furthermore, we showed that deep neural networks can be trained using a random sampling scheme that balances on multiple different labels simultaneously, without the need of storing patches on disk. Future work should involve more efficient patch generation and sampling, as well as improved clustering.

[1]  Andrew H. Beck,et al.  Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer , 2017, JAMA.

[2]  Thomas Rösch,et al.  Multi-scale fully convolutional neural networks for histopathology image segmentation: from nuclear aberrations to the global tissue architecture , 2019, Medical Image Anal..

[3]  Marco Novelli,et al.  Deep learning for prediction of colorectal cancer outcome: a discovery and validation study , 2020, The Lancet.

[4]  Ming Y. Lu,et al.  Data-efficient and weakly supervised computational pathology on whole-slide images , 2020, Nature Biomedical Engineering.

[5]  Li Wang,et al.  Segmentation of Craniomaxillofacial Bony Structures from MRI with a 3D Deep-Learning Based Cascade Framework , 2017, MLMI@MICCAI.

[6]  Ingerid Reinertsen,et al.  Meningioma Segmentation in T1-Weighted MRI Leveraging Global Context and Attention Mechanisms , 2021, Frontiers in Radiology.

[7]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Hong Liu,et al.  A Fast and Refined Cancer Regions Segmentation Framework in Whole-slide Breast Pathological Images , 2019, Scientific Reports.

[9]  Antonio J. Plaza,et al.  Image Segmentation Using Deep Learning: A Survey , 2021, IEEE transactions on pattern analysis and machine intelligence.

[10]  Shellaine R. Frazier,et al.  Cervical Whole Slide Histology Image Analysis Toolbox , 2020, medRxiv.

[11]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[12]  Max Welling,et al.  Attention-based Deep Multiple Instance Learning , 2018, ICML.

[13]  L. Vatten,et al.  Molecular subtypes, histopathological grade and survival in a historic cohort of breast cancer patients , 2013, Breast Cancer Research and Treatment.

[14]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  David B. A. Epstein,et al.  Fast and Accurate Tumor Segmentation of Histology Images using Persistent Homology and Deep Convolutional Features , 2018, Medical Image Anal..

[16]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[17]  Marcial García-Rojo,et al.  Automatic segmentation of whole-slide H&E stained breast histopathology images using a deep convolutional neural network architecture , 2020, Expert Syst. Appl..

[18]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[20]  I. Ellis,et al.  Pathological prognostic factors in breast cancer. I. The value of histological grade in breast cancer: experience from a large study with long-term follow-up. , 2002, Histopathology.

[21]  Joseph Redmon,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[22]  H. Bartsch,et al.  International Agency for Research on Cancer. , 1969, WHO chronicle.

[23]  Jianmin Li,et al.  Look Closer to Segment Better: Boundary Patch Refinement for Instance Segmentation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[25]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[26]  Jens Petersen,et al.  nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation , 2020, Nature Methods.

[27]  Ingerid Reinertsen,et al.  FastPathology: An Open-Source Platform for Deep Learning-Based Research and Decision Support in Digital Pathology , 2020, IEEE Access.

[28]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[29]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[30]  Michael A. Riegler,et al.  DoubleU-Net: A Deep Convolutional Neural Network for Medical Image Segmentation , 2020, 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS).

[31]  Aidan C. Daly,et al.  A convolutional neural network for common coordinate registration of high-resolution histology images , 2020, bioRxiv.

[32]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[33]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[34]  Bram van Ginneken,et al.  Resolution-agnostic tissue segmentation in whole-slide histopathology images with convolutional neural networks , 2019, PeerJ.

[35]  Stephen J. McKenna,et al.  Active Learning for Patch-Based Digital Pathology Using Convolutional Neural Networks to Reduce Annotation Costs , 2019, ECDP.

[36]  Brandon Ginley,et al.  Histo-fetch - On-the-fly processing of gigapixel whole slide images simplifies and speeds neural network training , 2022, Journal of Pathology Informatics.

[37]  J. S. Marron,et al.  A method for normalizing histology slides for quantitative analysis , 2009, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[38]  Geert J. S. Litjens,et al.  Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology , 2019, Medical Image Anal..

[39]  Zhuowen Tu,et al.  Multiple clustered instance learning for histopathology cancer image classification, segmentation and clustering , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Catarina Eloy,et al.  BACH: Grand Challenge on Breast Cancer Histology Images , 2018, Medical Image Anal..

[41]  Erik Smistad,et al.  High Performance Neural Network Inference, Streaming, and Visualization of Medical Images Using FAST , 2019, IEEE Access.

[42]  Peter Bankhead,et al.  QuPath: Open source software for digital pathology image analysis , 2017, Scientific Reports.

[43]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Mahadev Satyanarayanan,et al.  OpenSlide: A vendor-neutral software foundation for digital pathology , 2013, Journal of pathology informatics.

[45]  Anne L. Martel,et al.  Deep neural network models for computational histopathology: A survey , 2019, Medical Image Anal..

[46]  Kevin A. Schneider,et al.  Classification of Histopathological Biopsy Images Using Ensemble of Deep Learning Networks , 2019, CASCON.

[47]  Olivier Bernard,et al.  Cardiac MRI Segmentation with Strong Anatomical Guarantees , 2019, MICCAI.

[48]  Mehran Ebrahimi,et al.  Two-Stage Convolutional Neural Network for Breast Cancer Histology Image Classification , 2018, ICIAR.

[49]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[50]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[51]  Junzhou Huang,et al.  Deep Multi-instance Learning for Survival Prediction from Whole Slide Images , 2019, MICCAI.

[52]  Jitendra Jonnagaddala,et al.  Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks , 2020, Medical Image Anal..