HER2 Molecular Marker Scoring Using Transfer Learning and Decision Level Fusion

In prognostic evaluation of breast cancer, immunohistochemical (IHC) marker human epidermal growth factor receptor 2 (HER2) is used for prognostic evaluation. Accurate assessment of HER2-stained tissue sample is essential in therapeutic decision making for the patients. In regular clinical settings, expert pathologists assess the HER2-stained tissue slide under microscope for manual scoring based on prior experience. Manual scoring is time consuming, tedious, and often prone to inter-observer variation among group of pathologists. With the recent advancement in the area of computer vision and deep learning, medical image analysis has got significant attention. A number of deep learning architectures have been proposed for classification of different image groups. These networks are also used for transfer learning to classify other image classes. In the presented study, a number of transfer learning architectures are used for HER2 scoring. Five pre-trained architectures viz. VGG16, VGG19, ResNet50, MobileNetV2, and NASNetMobile with decimating the fully connected layers to get 3-class classification have been used for the comparative assessment of the networks as well as further scoring of stained tissue sample image based on statistical voting using mode operator. HER2 Challenge dataset from Warwick University is used in this study. A total of 2130 image patches were extracted to generate the training dataset from 300 training images corresponding to 30 training cases. The output model is then tested on 800 new test image patches from 100 test images acquired from 10 test cases (different from training cases) to report the outcome results. The transfer learning models have shown significant accuracy with VGG19 showing the best accuracy for the test images. The accuracy is found to be 93%, which increases to 98% on the image-based scoring using statistical voting mechanism. The output shows a capable quantification pipeline in automated HER2 score generation.

[1]  Alexei A. Efros,et al.  What makes ImageNet good for transfer learning? , 2016, ArXiv.

[2]  A. Ruifrok,et al.  Quantification of histochemical staining by color deconvolution. , 2001, Analytical and quantitative cytology and histology.

[3]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[4]  Ehsan Kazemi,et al.  Deep Convolutional Neural Networks Enable Discrimination of Heterogeneous Digital Pathology Images , 2017, bioRxiv.

[5]  Martin Tabakov,et al.  Segmentation of histopathology HER2/neu images with fuzzy decision tree and Takagi-Sugeno reasoning , 2014, Comput. Biol. Medicine.

[6]  John M S Bartlett,et al.  Recommendations for human epidermal growth factor receptor 2 testing in breast cancer: American Society of Clinical Oncology/College of American Pathologists clinical practice guideline update. , 2014, Archives of pathology & laboratory medicine.

[7]  Chandan Chakraborty,et al.  Her2Net: A Deep Framework for Semantic Segmentation and Classification of Cell Membranes and Nuclei in Breast Cancer Evaluation , 2018, IEEE Transactions on Image Processing.

[8]  Søren Nielsen,et al.  Digital image analysis of membrane connectivity is a robust measure of HER2 immunostains , 2012, Breast Cancer Research and Treatment.

[9]  Lynne Dobson,et al.  Image analysis as an adjunct to manual HER-2 immunohistochemical review: a diagnostic tool to standardize interpretation , 2010, Histopathology.

[10]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[11]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[12]  Stanislaw Osowski,et al.  Hourglass Shapes in Rank Grey-Level Hit-or-miss Transform for Membrane Segmentation in HER2/neu Images , 2015, ISMM.

[13]  Gustavo K. Rohde,et al.  Epithelium-Stroma Classification via Convolutional Neural Networks and Unsupervised Domain Adaptation in Histopathological Images , 2017, IEEE Journal of Biomedical and Health Informatics.

[14]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Ramakrishnan Mukundan,et al.  HER2 challenge contest: a detailed assessment of automated HER2 scoring algorithms in whole slide images of breast cancer tissues , 2017, Histopathology.

[16]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[17]  Kyle J. Myers,et al.  Automated evaluation of HER-2/neu immunohistochemical expression in breast cancer using digital microscopy , 2008, 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[18]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Sudipta Mukhopadhyay,et al.  AutoIHC‐Analyzer: computer‐assisted microscopy for automated membrane extraction/scoring in HER2 molecular markers , 2020, Journal of microscopy.

[20]  Jianhui Chen,et al.  Automated grading of breast cancer histopathology using cascaded ensemble with combination of multi-level image features , 2017, Neurocomputing.

[21]  Eslie Dennis,et al.  The assessment of HER2 status in breast cancer: the past, the present, and the future , 2016, Pathology international.

[22]  Nasir M. Rajpoot,et al.  Learning Where to See: A Novel Attention Model for Automated Immunohistochemical Scoring , 2019, IEEE Transactions on Medical Imaging.

[23]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Päivi Heikkilä,et al.  ER, PR, HER2, Ki-67 and CK5 in Early and Late Relapsing Breast Cancer—Reduced CK5 Expression in Metastases , 2013, Breast cancer : basic and clinical research.

[25]  Luca Maria Gambardella,et al.  Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks , 2013, MICCAI.

[26]  Joel J. P. C. Rodrigues,et al.  A novel deep learning based framework for the detection and classification of breast cancer using transfer learning , 2019, Pattern Recognit. Lett..

[27]  Michel E. Vandenberghe,et al.  Relevance of deep learning to facilitate the diagnosis of HER2 status in breast cancer , 2017, Scientific Reports.

[28]  Ramakrishnan Mukundan,et al.  Analysis of Image Feature Characteristics for Automated Scoring of HER2 in Histology Slides , 2019, J. Imaging.

[29]  Phillip M. Cheng,et al.  Transfer Learning with Convolutional Neural Networks for Classification of Abdominal Ultrasound Images , 2017, Journal of Digital Imaging.

[30]  Jorma Isola,et al.  ImmunoMembrane: a publicly available web application for digital image analysis of HER2 immunohistochemistry , 2012, Histopathology.

[31]  John M S Bartlett,et al.  Updated UK Recommendations for HER2 assessment in breast cancer , 2014, Journal of Clinical Pathology.

[32]  Jeovane H. Alves,et al.  An Automatic Patch-based Approach for HER-2 Scoring in Immunohistochemical Breast Cancer Images Using Color Features , 2018, Anais do Simpósio Brasileiro de Computação Aplicada à Saúde (SBCAS).

[33]  Hayden Kwok-Hay So,et al.  Large-Scale Multi-Class Image-Based Cell Classification With Deep Learning , 2019, IEEE Journal of Biomedical and Health Informatics.

[34]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Chen Liu,et al.  Quantitation of HER2/neu expression in primary gastroesophageal adenocarcinomas using conventional light microscopy and quantitative image analysis. , 2012, Archives of pathology & laboratory medicine.

[36]  Joerg Schwock,et al.  Quantitative image analysis of immunohistochemical stains using a CMYK color model , 2007, Diagnostic pathology.

[37]  Sos Agaian,et al.  Computer-Aided Prostate Cancer Diagnosis From Digitized Histopathology: A Review on Texture-Based Systems , 2015, IEEE Reviews in Biomedical Engineering.