Methods for Segmentation and Classification of Digital Microscopy Tissue Images

High-resolution microscopy images of tissue specimens provide detailed information about the morphology of normal and diseased tissue. Image analysis of tissue morphology can help cancer researchers develop a better understanding of cancer biology. Segmentation of nuclei and classification of tissue images are two common tasks in tissue image analysis. Development of accurate and efficient algorithms for these tasks is a challenging problem because of the complexity of tissue morphology and tumor heterogeneity. In this paper we present two computer algorithms; one designed for segmentation of nuclei and the other for classification of whole slide tissue images. The segmentation algorithm implements a multiscale deep residual aggregation network to accurately segment nuclear material and then separate clumped nuclei into individual nuclei. The classification algorithm initially carries out patch-level classification via a deep learning method, then patch-level statistical and morphological features are used as input to a random forest regression model for whole slide image classification. The segmentation and classification algorithms were evaluated in the MICCAI 2017 Digital Pathology challenge. The segmentation algorithm achieved an accuracy score of 0.78. The classification algorithm achieved an accuracy score of 0.81. These scores were the highest in the challenge.

[1]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[2]  Lin Yang,et al.  Transfer Shape Modeling Towards High-Throughput Microscopy Image Segmentation , 2016, MICCAI.

[3]  Alejandro F. Frangi,et al.  Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015 , 2015, Lecture Notes in Computer Science.

[4]  Adel Hafiane,et al.  Integrating segmentation with deep learning for enhanced classification of epithelial and stromal tissues in H&E images , 2017, Pattern Recognit. Lett..

[5]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[6]  Nasir M. Rajpoot,et al.  Locality Sensitive Deep Learning for Detection and Classification of Nuclei in Routine Colon Cancer Histology Images , 2016, IEEE Trans. Medical Imaging.

[7]  Christoph Meinel,et al.  Deep Learning for Medical Image Analysis , 2018, Journal of Pathology Informatics.

[8]  Hao Chen,et al.  DCAN: Deep contour‐aware networks for object instance segmentation from histology images , 2017, Medical Image Anal..

[9]  Filip Karlo Dosilovic,et al.  Explainable artificial intelligence: A survey , 2018, 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).

[10]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Joel H. Saltz,et al.  Unsupervised Histopathology Image Synthesis , 2017, ArXiv.

[13]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  A. Madabhushi.  Digital pathology image analysis: opportunities and challenges. , 2009, Imaging in medicine.

[15]  Brian Vastag.  Panel backs new NIH center devoted to translational medicine , 2011, Nature Medicine.

[16]  Hai Su,et al.  Deep Voting: A Robust Approach Toward Nucleus Localization in Microscopy Images , 2015, MICCAI.

[17]  L P Clarke,et al.  An Assessment of Imaging Informatics for Precision Medicine in Cancer , 2017, Yearbook of Medical Informatics.

[18]  Andreas Holzinger,et al.  Machine Learning for Health Informatics , 2016, Lecture Notes in Computer Science.

[19]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[20]  Andreas Nürnberger,et al.  The Power of Ensembles for Active Learning in Image Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  M. Stella Atkins,et al.  How Users Perceive Content-Based Image Retrieval for Identifying Skin Images , 2018, MLCN/DLF/iMIMIC@MICCAI.

[22]  Jitendra Malik,et al.  Hypercolumns for object segmentation and fine-grained localization , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Joel H. Saltz,et al.  Patch-Based Convolutional Neural Network for Whole Slide Tissue Image Classification , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Anne L. Martel,et al.  Automatic cell detection and segmentation from H and E stained pathology slides using colorspace decorrelation stretching , 2016, SPIE Medical Imaging.

[25]  Andreas Holzinger,et al.  Interactive machine learning for health informatics: when do we need the human-in-the-loop? , 2016, Brain Informatics.

[26]  Nasir M. Rajpoot,et al.  SAMS-NET: Stain-aware multi-scale network for instance-based nuclei segmentation in histology images , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[27]  Lin Yang,et al.  Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review , 2016, IEEE Reviews in Biomedical Engineering.

[28]  Andrew Evans,et al.  Digital imaging in pathology: whole-slide imaging and beyond. , 2013, Annual review of pathology.

[29]  Richard J. Chen,et al.  Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images , 2018, IEEE Transactions on Medical Imaging.

[30]  Fang Zhang,et al.  Deep convolutional activation features for large scale Brain Tumor histopathology image classification and segmentation , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[31]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[33]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[34]  A. Madabhushi,et al.  Histopathological Image Analysis: A Review , 2009, IEEE Reviews in Biomedical Engineering.

[35]  Zhiguo Jiang,et al.  Feature extraction from histopathological images based on nucleus-guided convolutional neural network for breast lesion classification , 2017, Pattern Recognit..

[36]  L. R. Dice.  Measures of the Amount of Ecologic Association Between Species , 1945 .

[37]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Joel H Saltz,et al.  PanCancer insights from The Cancer Genome Atlas: the pathologist's perspective , 2018, The Journal of pathology.

[39]  Lin Yang,et al.  Spatial Clockwork Recurrent Neural Network for Muscle Perimysium Segmentation , 2016, MICCAI.

[40]  Anne L. Martel,et al.  A Cluster-then-label Semi-supervised Learning Approach for Pathology Image Classification , 2018, Scientific Reports.

[41]  Andreas Holzinger,et al.  Interactive machine learning: experimental evidence for the human in the algorithmic loop , 2018, Applied Intelligence.

[42]  Ce Zhang,et al.  Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features , 2016, Nature Communications.

[43]  Nasir M. Rajpoot,et al.  Representation-Aggregation Networks for Segmentation of Multi-Gigapixel Histology Images , 2017, ArXiv.

[44]  Junzhou Huang,et al.  Subtype Cell Detection with an Accelerated Deep Convolution Neural Network , 2016, MICCAI.

[45]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  N. Razavian,et al.  Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning , 2018, Nature Medicine.

[47]  Joel H. Saltz,et al.  A Methodology for Texture Feature-based Quality Assessment in Nucleus Segmentation of Histopathology Image , 2017, Journal of pathology informatics.

[48]  Nico Karssemeijer,et al.  Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images , 2017, Journal of medical imaging.

[49]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[50]  Stephen J. McKenna,et al.  An automated pattern recognition system for classifying indirect immunofluorescence images of HEp-2 cells and specimens , 2016, Pattern Recognit..

[51]  Vishal Monga,et al.  Histopathological Image Classification Using Discriminative Feature-Oriented Dictionary Learning , 2015, IEEE Transactions on Medical Imaging.

[52]  Andreas Holzinger,et al.  From Machine Learning to Explainable AI , 2018, 2018 World Symposium on Digital Intelligence for Systems and Machines (DISA).