Experimental Assessment of Color Deconvolution and Color Normalization for Automated Classification of Histology Images Stained with Hematoxylin and Eosin

Simple Summary The appearance of histology images stained with H&E can vary a lot as a consequence of changes in the reagents, staining conditions, preparation procedure and acquisition system. In this work we investigated whether color preprocessing—specifically color deconvolution and color normalization—could be used to correct such variability and improve the performance of automated classification procedures. Experimenting on 11 datasets, 13 image descriptors and eight color pre-processing methods we found that doing no color preprocessing was the best option in most cases. Abstract Histological evaluation plays a major role in cancer diagnosis and treatment. The appearance of H&E-stained images can vary significantly as a consequence of differences in several factors, such as reagents, staining conditions, preparation procedure and image acquisition system. Such potential sources of noise can all have negative effects on computer-assisted classification. To minimize such artefacts and their potentially negative effects several color pre-processing methods have been proposed in the literature—for instance, color augmentation, color constancy, color deconvolution and color transfer. Still, little work has been done to investigate the efficacy of these methods on a quantitative basis. In this paper, we evaluated the effects of color constancy, deconvolution and transfer on automated classification of H&E-stained images representing different types of cancers—specifically breast, prostate, colorectal cancer and malignant lymphoma. Our results indicate that in most cases color pre-processing does not improve the classification accuracy, especially when coupled with color-based image descriptors. Some pre-processing methods, however, can be beneficial when used with some texture-based methods like Gabor filters and Local Binary Patterns.

[1]  Andrew Janowczyk,et al.  Stain Normalization using Sparse AutoEncoders (StaNoSA): Application to digital pathology , 2017, Comput. Medical Imaging Graph..

[2]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[3]  Paolo Napoletano,et al.  Hand-Crafted vs Learned Descriptors for Color Texture Classification , 2017, CCIW.

[4]  D. Foster Color constancy , 2011, Vision Research.

[5]  J. S. Marron,et al.  A method for normalizing histology slides for quantitative analysis , 2009, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[6]  Nasir M. Rajpoot,et al.  A Nonlinear Mapping Approach to Stain Normalization in Digital Histopathology Images Using Image-Specific Color Deconvolution , 2014, IEEE Transactions on Biomedical Engineering.

[7]  Joost van de Weijer,et al.  Color in Computer Vision , 2008 .

[8]  Geert J. S. Litjens,et al.  Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology , 2019, Medical Image Anal..

[9]  Jin Tae Kwak,et al.  Multiview boosting digital pathology analysis of prostate cancer , 2017, Comput. Methods Programs Biomed..

[10]  D. Brat,et al.  Predicting cancer outcomes from histology and genomics using convolutional networks , 2017, Proceedings of the National Academy of Sciences.

[11]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Fabrizio Smeraldi,et al.  Comparative Evaluation of Hand-Crafted Image Descriptors vs. Off-the-Shelf CNN-Based Features for Colour Texture Classification under Ideal and Realistic Conditions , 2019, Applied Sciences.

[13]  Hieu T. Nguyen,et al.  Deep Learning Applied for Histological Diagnosis of Breast Cancer , 2020, IEEE Access.

[14]  Matti Pietikäinen,et al.  Identification of tumor epithelium and stroma in tissue microarrays using texture analysis , 2012, Diagnostic Pathology.

[15]  Anant Madabhushi,et al.  Statistical shape model for manifold regularization: Gleason grading of prostate histology , 2013, Comput. Vis. Image Underst..

[16]  Francesco Bianconi,et al.  Collection of textures in colorectal cancer histology , 2016 .

[17]  Sidra Nawaz,et al.  Mapping spatial heterogeneity in the tumor microenvironment: a new era for digital pathology , 2015, Laboratory Investigation.

[18]  Giuseppe De Pietro,et al.  A Machine-learning Approach for the Assessment of the Proliferative Compartment of Solid Tumors on Hematoxylin-Eosin-Stained Sections , 2020, Cancers.

[19]  Nassir Navab,et al.  Structure-Preserving Color Normalization and Sparse Stain Separation for Histological Images , 2016, IEEE Transactions on Medical Imaging.

[20]  A. Madabhushi,et al.  Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology , 2019, Nature Reviews Clinical Oncology.

[21]  Ziqian Wu,et al.  A machine learning-based prognostic predictor for stage III colon cancer , 2020, Scientific Reports.

[22]  Lasse Riis Østergaard,et al.  Exploiting Multiple Color Representations to Improve Colon Cancer Detection in Whole Slide H&E Stains , 2018, COMPAY/OMIA@MICCAI.

[23]  Constantino Carlos Reyes-Aldasoro,et al.  Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study , 2019, PLoS medicine.

[24]  Dmitrii Bychkov,et al.  Deep learning based tissue analysis predicts outcome in colorectal cancer , 2018, Scientific Reports.

[25]  Andrea Vedaldi,et al.  MatConvNet: Convolutional Neural Networks for MATLAB , 2014, ACM Multimedia.

[26]  Nasir M. Rajpoot,et al.  A Novel Digital Score for Abundance of Tumour Infiltrating Lymphocytes Predicts Disease Free Survival in Oral Squamous Cell Carcinoma , 2019, Scientific Reports.

[27]  Konstantinos N. Plataniotis,et al.  Circular Mixture Modeling of Color Distribution for Blind Stain Separation in Pathology Images , 2017, IEEE Journal of Biomedical and Health Informatics.

[28]  Hao Chen,et al.  Gland segmentation in colon histology images: The glas challenge contest , 2016, Medical Image Anal..

[29]  Manasi Gyanchandani,et al.  Machine Learning Methods for Computer-Aided Breast Cancer Diagnosis Using Histopathology: A Narrative Review. , 2019, Journal of medical imaging and radiation sciences.

[30]  Darren Treanor,et al.  Digital pathology access and usage in the UK: results from a national survey on behalf of the National Cancer Research Institute’s CM-Path initiative , 2018, Journal of Clinical Pathology.

[31]  Stanislav Kovacic,et al.  Rotation-invariant texture classification , 2003, Pattern Recognit. Lett..

[32]  Peter Hufnagl,et al.  Computational morphogenesis - Embryogenesis, cancer research and digital pathology , 2018, Biosyst..

[33]  Shyam Lal,et al.  A study about color normalization methods for histopathology images. , 2018, Micron.

[34]  Francesco Bianconi,et al.  Multi-class texture analysis in colorectal cancer histology , 2016, Scientific Reports.

[35]  Paolo Napoletano,et al.  Improved opponent color local binary patterns: an effective local image descriptor for color texture classification , 2017, J. Electronic Imaging.

[36]  Neeraj Kumar,et al.  Empirical comparison of color normalization methods for epithelial-stromal classification in H and E images , 2016, Journal of pathology informatics.

[37]  Constantino Carlos Reyes-Aldasoro,et al.  Evaluation of Colour Pre-processing on Patch-Based Classification of H&E-Stained Images , 2019, ECDP.

[38]  Peter H. N. de With,et al.  Stain normalization of histopathology images using generative adversarial networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[39]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[40]  Paolo Napoletano,et al.  Evaluating color texture descriptors under large variations of controlled lighting conditions , 2015, Journal of the Optical Society of America. A, Optics, image science, and vision.

[41]  A. Ruifrok,et al.  Quantification of histochemical staining by color deconvolution. , 2001, Analytical and quantitative cytology and histology.

[42]  Graham D. Finlayson,et al.  Colour indexing across devices and viewing conditions , 2001 .

[43]  Nasir M. Rajpoot,et al.  A Stochastic Polygons Model for Glandular Structures in Colon Histology Images , 2015, IEEE Transactions on Medical Imaging.

[44]  Luiz Eduardo Soares de Oliveira,et al.  A Dataset for Breast Cancer Histopathological Image Classification , 2016, IEEE Transactions on Biomedical Engineering.

[45]  D. Cavouras,et al.  Computer-based association of the texture of expressed estrogen receptor nuclei with histologic grade using immunohistochemically-stained breast carcinomas. , 2009, Analytical and quantitative cytology and histology.

[46]  Nassir Navab,et al.  Staingan: Stain Style Transfer for Digital Histological Images , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[47]  Begonya Garcia-Zapirain,et al.  Breast Cancer Histopathology Image Classification Using an Ensemble of Deep Learning Models , 2020, Sensors.

[48]  Arkadiusz Gertych,et al.  Machine learning approaches to analyze histological images of tissues from radical prostatectomies , 2015, Comput. Medical Imaging Graph..

[49]  Francesco Bianconi,et al.  Automatic Characterization of the Visual Appearance of Industrial Materials through Colour and Texture Analysis: An Overview of Methods and Applications , 2013 .

[50]  Francesco Bianconi,et al.  Rotation invariant co-occurrence features based on digital circles and discrete Fourier transform , 2014, Pattern Recognit. Lett..

[51]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[53]  A. Madabhushi Digital pathology image analysis: opportunities and challenges. , 2009, Imaging in medicine.

[54]  Andrew H. Beck,et al.  Systematic Analysis of Breast Cancer Morphology Uncovers Stromal Features Associated with Survival , 2011, Science Translational Medicine.

[55]  Jens Rittscher,et al.  The use of digital pathology and image analysis in clinical trials , 2019, The journal of pathology. Clinical research.

[56]  Nikos Grammalidis,et al.  Grading of invasive breast carcinoma through Grassmannian VLAD encoding , 2017, PloS one.

[57]  T. Meckel,et al.  A Model based Survey of Colour Deconvolution in Diagnostic Brightfield Microscopy: Error Estimation and Spectral Consideration , 2015, Scientific Reports.

[58]  Andrew Zisserman,et al.  Return of the Devil in the Details: Delving Deep into Convolutional Nets , 2014, BMVC.

[59]  J. Slodkowska,et al.  Digital pathology in personalized cancer therapy. , 2012, Folia histochemica et cytobiologica.

[60]  Xuejie Zhang,et al.  Parallel Structure Deep Neural Network Using CNN and RNN with an Attention Mechanism for Breast Cancer Histology Image Classification , 2019, Cancers.

[61]  Bram van Ginneken,et al.  The importance of stain normalization in colorectal tissue classification with convolutional networks , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[62]  Matti Pietikäinen,et al.  Accurate color discrimination with classification based on feature distributions , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[63]  Michael Gadermayr,et al.  A Quantitative Assessment of Image Normalization for Classifying Histopathological Tissue of the Kidney , 2017, GCPR.

[64]  Manuel Fernández Delgado,et al.  Influence of normalization and color space to color texture classification , 2017, Pattern Recognit..

[65]  Marco Novelli,et al.  Deep learning for prediction of colorectal cancer outcome: a discovery and validation study , 2020, The Lancet.

[66]  Luiz Eduardo Soares de Oliveira,et al.  Multiple instance learning for histopathological breast cancer image classification , 2019, Expert Syst. Appl..

[67]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[68]  Daisuke Komura,et al.  Machine Learning Methods for Histopathological Image Analysis , 2017, Computational and structural biotechnology journal.

[69]  Lior Shamir,et al.  IICBU 2008: a proposed benchmark suite for biological image analysis , 2008, Medical & Biological Engineering & Computing.

[70]  Jon Griffin,et al.  Digital pathology in clinical use: where are we now and what is holding us back? , 2017, Histopathology.

[71]  Darren Treanor,et al.  Colour in digital pathology: a review , 2017, Histopathology.