GGAC: Multi-relational image gated GCN with attention convolutional binary neural tree for identifying disease with chest X-rays

Abstract Using medical images for disease identification is an important application in the medical field. Graph Convolutional Network (GCN) is proposed to model multi-relational image and generate more informative image representations. Recently, the relations between medical images are utilized to identify diseases. This paper proposes a Gated GCN with Attention Convolutional Binary Neural Tree (GGAC) for Multi-Relational Image Identifying Disease. GGAC extracts the discriminative features of the image, strengthen the ability to model medical images, understands images representation deeply and then well captures the multi-modal relation between images. Firstly, an Attention Convolutional Binary Neural Tree based on the attention mechanism is designed to extract the fine-grained features of the images, and use the attention conversion operation on the edge of the tree structure to enhance the network’s acquisition of key image features. Secondly, a Gated GCN is proposed to improve GCN performance by solving the problem of the weight distribution of different neighbors in the same-order neighborhood. Thirdly, a GCN propagation rule is used to transfer messages in multi-relational Graph and then solves the message passing problem of high-dimensional feature data in GCN. Finally, we verify GGAC on a multi-relational graph constructed on the Chest X-rays14. It can be seen from the experiment that overfitting and underfitting can be solved to a certain extent through the extraction and inference of the features of the multi-relational graph, and then GGAC has better performance than the state-of-the-art methods, and keeps good in model complexity.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Yuan Luo,et al.  Graph Convolutional Networks for Text Classification , 2018, AAAI.

[3]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Li Yao,et al.  Learning to diagnose from scratch by exploiting dependencies among labels , 2017, ArXiv.

[5]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[6]  Maoguo Gong,et al.  Graph convolutional networks with multi-level coarsening for graph classification , 2020, Knowl. Based Syst..

[7]  Xiao-Ming Wu,et al.  Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning , 2018, AAAI.

[8]  Cheng Wang,et al.  GMAN: A Graph Multi-Attention Network for Traffic Prediction , 2019, AAAI.

[9]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[10]  Yu-Chiang Frank Wang,et al.  Multi-label Zero-Shot Learning with Structured Knowledge Graphs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  Muktabh Mayank Srivastava,et al.  Boosted Cascaded Convnets for Multilabel Classification of Thoracic Diseases in Chest Radiographs , 2017, ICIAR.

[12]  Wei Wei,et al.  Thoracic Disease Identification and Localization with Limited Supervision , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[14]  Jonathan Masci,et al.  Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Shuqiang Jiang,et al.  Deep Structured Learning for Visual Relationship Detection , 2018, AAAI.

[16]  Yann Dauphin,et al.  Language Modeling with Gated Convolutional Networks , 2016, ICML.

[17]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[18]  Sarah Parisot,et al.  Learning Conditioned Graph Structures for Interpretable Visual Question Answering , 2018, NeurIPS.

[19]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[20]  Pierre Vandergheynst,et al.  Geometric Deep Learning: Going beyond Euclidean data , 2016, IEEE Signal Process. Mag..

[21]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Joan Bruna,et al.  Few-Shot Learning with Graph Neural Networks , 2017, ICLR.

[23]  Ronald M. Summers,et al.  ChestX-ray: Hospital-Scale Chest X-ray Database and Benchmarks on Weakly Supervised Classification and Localization of Common Thorax Diseases , 2019, Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics.

[24]  Abhinav Gupta,et al.  The More You Know: Using Knowledge Graphs for Image Classification , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Stella X. Yu,et al.  Affinity CNN: Learning Pixel-Centric Pairwise Relations for Figure/Ground Embedding , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Mei Xie,et al.  Identifying critical nodes in complex networks via graph convolutional networks , 2020, Knowl. Based Syst..

[27]  Yuan Luo,et al.  Deep Generative Classifiers for Thoracic Disease Diagnosis with Chest X-ray Images , 2018, 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[28]  Abhinav Gupta,et al.  Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Ning Feng,et al.  Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting , 2019, AAAI.

[30]  Bo Dai,et al.  Detecting Visual Relationships with Deep Relational Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Tao Mei,et al.  Exploring Visual Relationship for Image Captioning , 2018, ECCV.

[32]  Lanfen Lin,et al.  Medical Image Classification Using Deep Learning , 2019 .

[33]  Yuxing Tang,et al.  Attention-Guided Curriculum Learning for Weakly Supervised Classification and Localization of Thoracic Diseases on Chest Radiographs , 2018, MLMI@MICCAI.

[34]  Miguel Ángel González Ballester,et al.  Medical Image Segmentation Using Deep Learning , 2019, Intelligent Systems Reference Library.

[35]  Yaping Huang,et al.  Multi-label chest X-ray image classification via category-wise residual attention learning , 2020, Pattern Recognit. Lett..

[36]  Christopher D. Manning,et al.  Graph Convolution over Pruned Dependency Trees Improves Relation Extraction , 2018, EMNLP.

[37]  Chris H. Q. Ding,et al.  Image annotation using bi-relational graph of images and semantic labels , 2011, CVPR 2011.

[38]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[39]  Yan Shen,et al.  Dynamic Routing on Deep Neural Network for Thoracic Disease Classification and Sensitive Area Localization , 2018, MLMI@MICCAI.

[40]  Xiu-Shen Wei,et al.  Multi-Label Image Recognition With Graph Convolutional Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Yuan Luo,et al.  ImageGCN: Multi-Relational Image Graph Convolutional Networks for Disease Identification With Chest X-Rays , 2019, IEEE Transactions on Medical Imaging.

[42]  Amin Zheng,et al.  RGCNN: Regularized Graph CNN for Point Cloud Segmentation , 2018, ACM Multimedia.

[43]  Miguel Ángel González Ballester,et al.  Medical Image Detection Using Deep Learning , 2019 .

[44]  Xiaohan Wang,et al.  Symbiotic Attention for Egocentric Action Recognition With Object-Centric Alignment , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.