A hierarchical conditional random field-based attention mechanism approach for gastric histopathology image classification

In the Gastric Histopathology Image Classification (GHIC) tasks, which is usually weakly supervised learning missions, there is inevitably redundant information in the images. Therefore, designing networks that can focus on effective distinguishing features has become a popular research topic. In this paper, to accomplish the tasks of GHIC superiorly and to assist pathologists in clinical Yixin Li Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, China E-mail: 1047792668@qq.com Xinran Wu (co-frist author) Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, China Chen Li (corresponding author) Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, China E-mail: lichen201096@hotmail.com Changhao Sun Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, China; Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China Md Rahaman Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, China Yudong Yao Department of Electrical and Computer Engineering, Stevens Institute of Technology, Hoboken, NJ 07030, USA Xiaoyan Li China Medical University, Liaoning Cancer Hospital and Institute, Shenyang Yong Zhang China Medical University, Liaoning Cancer Hospital and Institute, Shenyang Tao Jiang Control Engineering College, Chengdu University of Information Technology, Chengdu People’s Republic of China ar X iv :2 10 2. 10 49 9v 1 [ cs .C V ] 2 1 Fe b 20 21

[1]  Daniel S. Kermany,et al.  Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning , 2018, Cell.

[2]  Chen Li,et al.  A Comprehensive Review for MRF and CRF Approaches in Pathology Image Analysis , 2020, ArXiv.

[3]  Kejun Wang,et al.  Human behavior recognition based on fractal conditional random field , 2013, 2013 25th Chinese Control and Decision Conference (CCDC).

[4]  K. Arihiro,et al.  Deep Learning Models for Histopathological Classification of Gastric and Colonic Epithelial Tumours , 2020, Scientific Reports.

[5]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[6]  Gang Zou,et al.  Single polydiacetylene microtube waveguide platform for discriminating microRNA-215 expression levels in clinical gastric cancerous, paracancerous and normal tissues. , 2018, Talanta.

[7]  Le Lu,et al.  Thorax-Net: An Attention Regularized Deep Neural Network for Classification of Thoracic Diseases on Chest Radiography , 2020, IEEE Journal of Biomedical and Health Informatics.

[8]  Guoli Wang,et al.  GECNN-CRF for Prostate Cancer Detection with WSI , 2020 .

[9]  Xiaofei Wang,et al.  Attention Based Glaucoma Detection: A Large-Scale Database and CNN Model , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Di Zhao,et al.  基于深度学习的胃癌病理图像分类方法 (Pathological Image Classification of Gastric Cancer Based on Depth Learning) , 2018, 计算机科学.

[11]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[12]  A. Jemal,et al.  Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries , 2018, CA: a cancer journal for clinicians.

[13]  Shotaro Akaho,et al.  TrBagg: A Simple Transfer Learning Method and its Application to Personalization in Collaborative Tagging , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[14]  J. Lasota,et al.  Gastrointestinal stromal tumors: review on morphology, molecular pathology, prognosis, and differential diagnosis. , 2009, Archives of pathology & laboratory medicine.

[15]  Yanping Zhang,et al.  Pyramid feature adaptation for semi-supervised cardiac bi-ventricle segmentation , 2020, Comput. Medical Imaging Graph..

[16]  Chen Li,et al.  Environmental microorganism classification using conditional random fields and deep convolutional neural networks , 2018, Pattern Recognit..

[17]  Sung-Hyon Myaeng,et al.  Toward advice mining: conditional random fields for extracting advice-revealing text units , 2013, CIKM.

[18]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Mohamed Hammad,et al.  ResNet‐Attention model for human authentication using ECG signals , 2020, Expert Syst. J. Knowl. Eng..

[20]  Stephen Lin,et al.  GCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[21]  Joel H. Saltz,et al.  Methods for Segmentation and Classification of Digital Microscopy Tissue Images , 2018, Front. Bioeng. Biotechnol..

[22]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Pheng-Ann Heng,et al.  RMDL: Recalibrated multi-instance deep learning for whole slide gastric image classification , 2019, Medical Image Anal..

[24]  Wenlong Feng,et al.  Automated Gleason Grading and Gleason Pattern Region Segmentation Based on Deep Learning for Pathological Images of Prostate Cancer , 2020, IEEE Access.

[25]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[26]  L. Bruzzone,et al.  Attention-Based Adaptive Spectral–Spatial Kernel ResNet for Hyperspectral Image Classification , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[27]  Chao Di,et al.  U1 snRNP regulates cancer cell migration and invasion in vitro , 2020, Nature Communications.

[28]  Hamidullah Binol,et al.  Classification of molecular structure images by using ANN, RF, LBP, HOG, and size reduction methods for early stomach cancer detection , 2018 .

[29]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Fei Cheng,et al.  Object detection based on an adaptive attention mechanism , 2020, Scientific Reports.

[31]  Jing Li,et al.  Detection of gastric cancer and its histological type based on iodine concentration in spectral CT , 2018, Cancer Imaging.

[32]  Gorjan Alagic,et al.  #p , 2019, Quantum information & computation.

[33]  Ling Hong,et al.  A local mean and variance active contour model for biomedical image segmentation , 2019, J. Comput. Sci..

[34]  Mingon Kang,et al.  Deep-Hipo: Multi-scale Receptive Field Deep Learning for Histopathological Image Analysis. , 2020, Methods.

[35]  P. Alam,et al.  H , 1887, High Explosives, Propellants, Pyrotechnics.

[36]  Olaf Hellwich,et al.  Appearance-based necrosis detection using textural features and SVM with discriminative thresholding in histopathological whole slide images , 2015, 2015 IEEE 15th International Conference on Bioinformatics and Bioengineering (BIBE).

[37]  A. Renshaw,et al.  American society of cytopathology workload recommendations for automated pap test screening: Developed by the productivity and quality assurance in the era of automated screening task force , 2013, Diagnostic cytopathology.

[38]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[39]  Ning Xu,et al.  Cervical Histopathology Image Classification Using Multilayer Hidden Conditional Random Fields and Weakly Supervised Learning , 2019, IEEE Access.

[40]  Yanjie Wei,et al.  miRNA‐192 and ‐215 activate Wnt/β‐catenin signaling pathway in gastric cancer via APC , 2020, Journal of cellular physiology.

[41]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Jiri Matas,et al.  All you need is a good init , 2015, ICLR.

[43]  Srinivas S. Kruthiventi,et al.  Crowd flow segmentation in compressed domain using CRF , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[44]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  John Thickstun,et al.  CONDITIONAL RANDOM FIELDS , 2016 .

[46]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[47]  Ming Zhang,et al.  Classification of gastric slices based on deep learning and sparse representation , 2018, 2018 Chinese Control And Decision Conference (CCDC).

[48]  Ioannis Roxanis,et al.  Superpixel-Based Conditional Random Fields (SuperCRF): Incorporating Global and Local Context for Enhanced Deep Learning in Melanoma Histopathology , 2019, Front. Oncol..

[49]  Heechan Yang,et al.  Guided Soft Attention Network for Classification of Breast Cancer Histopathology Images , 2020, IEEE Transactions on Medical Imaging.

[50]  Nima Tajbakhsh,et al.  UNet++: A Nested U-Net Architecture for Medical Image Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[51]  M. Mildner,et al.  Re-epithelialization and immune cell behaviour in an ex vivo human skin model , 2020, Scientific Reports.

[52]  Chen Li,et al.  Gastric histopathology image segmentation using a hierarchical conditional random field , 2020, Biocybernetics and Biomedical Engineering.

[53]  Hao Chen,et al.  Weakly Supervised Cervical Histopathological Image Classification Using Multilayer Hidden Conditional Random Fields , 2019, ITIB.

[54]  Tao Xu,et al.  Computer-Aided Diagnosis in Histopathological Images of the Endometrium Using a Convolutional Neural Network and Attention Mechanisms , 2019, IEEE Journal of Biomedical and Health Informatics.

[55]  Luc Vincent,et al.  Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[56]  William Speier,et al.  A Multi-resolution Model for Histopathology Image Classification and Localization with Multiple Instance Learning , 2020, Comput. Biol. Medicine.

[57]  Miki Haseyama,et al.  Detection of gastric cancer risk from X-ray images via patch-based convolutional neural network , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[58]  Weili Guan,et al.  Image caption generation with dual attention mechanism , 2020, Inf. Process. Manag..

[59]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[60]  Hamidullah Binol,et al.  Recognition of the stomach cancer images with probabilistic HOG feature vector histograms by using HOG features , 2017, 2017 IEEE 15th International Symposium on Intelligent Systems and Informatics (SISY).

[61]  Martial Hebert,et al.  Discriminative Random Fields , 2006, International Journal of Computer Vision.

[62]  Rong Zhang,et al.  Lesion detection of endoscopy images based on convolutional neural network features , 2015, 2015 8th International Congress on Image and Signal Processing (CISP).

[63]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[64]  B. Stewart,et al.  World cancer report 2014. , 2014 .

[65]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[66]  Heikki Haario,et al.  Segmentation of Overlapping Elliptical Objects in Silhouette Images , 2015, IEEE Transactions on Image Processing.

[67]  Xiao Zhang,et al.  Semi-supervised Structured Prediction with Neural CRF Autoencoder , 2017, EMNLP.

[68]  Peter Clifford,et al.  Markov Random Fields in Statistics , 2012 .

[69]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[70]  C. Jaspin Jeba Sheela,et al.  Morphological edge detection and brain tumor segmentation in Magnetic Resonance (MR) images based on region growing and performance evaluation of modified Fuzzy C-Means (FCM) algorithm , 2020, Multimedia Tools and Applications.

[71]  Stan Z. Li,et al.  Markov Random Field Models in Computer Vision , 1994, ECCV.

[72]  Ruslan Salakhutdinov,et al.  Action Recognition using Visual Attention , 2015, NIPS 2015.

[73]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  J. Lasota,et al.  Gastrointestinal stromal tumors (GISTs): definition, occurrence, pathology, differential diagnosis and molecular genetics. , 2003, Polish journal of pathology : official journal of the Polish Society of Pathologists.

[75]  Chen Li,et al.  Hierarchical conditional random field model for multi‐object segmentation in gastric histopathology images , 2020 .

[76]  Zhenzhou Wang,et al.  A semi-automatic method for robust and efficient identification of neighboring muscle cells , 2016, Pattern Recognit..

[77]  Ghassan Hamarneh,et al.  Predicting Cancer with a Recurrent Visual Attention Model for Histopathology Images , 2018, MICCAI.

[78]  Koray Kavukcuoglu,et al.  Visual Attention , 2020, Computational Models for Cognitive Vision.

[79]  Olaf Hellwich,et al.  A Comparative Study of Cell Nuclei Attributed Relational Graphs for Knowledge Description and Categorization in Histopathological Gastric Cancer Whole Slide Images , 2017, 2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS).

[80]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[81]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[82]  Asoke K. Nandi,et al.  Significantly Fast and Robust Fuzzy C-Means Clustering Algorithm Based on Morphological Reconstruction and Membership Filtering , 2018, IEEE Transactions on Fuzzy Systems.

[83]  Olaf Hellwich,et al.  Deep convolutional neural networks for automatic classification of gastric carcinoma using whole slide images in digital histopathology , 2017, Comput. Medical Imaging Graph..

[84]  J. Sethian,et al.  Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations , 1988 .

[85]  F. Ciardiello,et al.  Treatment of gastric cancer. , 2014, World journal of gastroenterology.

[86]  Chen Li,et al.  Intelligent Gastric Histopathology Image Classification Using Hierarchical Conditional Random Field based Attention Mechanism , 2021, ICMLC.

[87]  Yashwant Kurmi,et al.  Content-based image retrieval algorithm for nuclei segmentation in histopathology images , 2020, Multimedia Tools and Applications.

[88]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[89]  Dimitris N. Metaxas,et al.  Weakly Supervised Deep Nuclei Segmentation using Points Annotation in Histopathology Images , 2019, MIDL.

[90]  Xiaofei Wang,et al.  A Large-Scale Database and a CNN Model for Attention-Based Glaucoma Detection , 2020, IEEE Transactions on Medical Imaging.

[91]  A. Fischer,et al.  Hematoxylin and eosin staining of tissue and cell sections. , 2008, CSH protocols.

[92]  Chan Basaruddin,et al.  A review on conditional random fields as a sequential classifier in machine learning , 2017, 2017 International Conference on Electrical Engineering and Computer Science (ICECOS).

[93]  Rong Li,et al.  Gastric Pathology Image Recognition Based on Deep Residual Networks , 2018, 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC).

[94]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[95]  Richard C. Davis,et al.  Clinically applicable histopathological diagnosis system for gastric cancer detection using deep learning , 2020, Nature Communications.

[96]  H. Espejo,et al.  [Gastric cancer]. , 1996, Revista de gastroenterologia del Peru : organo oficial de la Sociedad de Gastroenterologia del Peru.

[97]  Y. Shibamoto,et al.  Identification of the pericardiacophrenic vein on CT , 2018, Cancer Imaging.