Structural Attention Graph Neural Network for Diagnosis and Prediction of COVID-19 Severity

With rapid worldwide spread of Coronavirus Disease 2019 (COVID-19), jointly identifying severe COVID-19 cases from mild ones and predicting the conversion time (from mild to severe) is essential to optimize the workflow and reduce the clinician’s workload. In this study, we propose a novel framework for COVID-19 diagnosis, termed as Structural Attention Graph Neural Network (SAGNN), which can combine the multi-source information including features extracted from chest CT, latent lung structural distribution, and non-imaging patient information to conduct diagnosis of COVID-19 severity and predict the conversion time from mild to severe. Specifically, we first construct a graph to incorporate structural information of the lung and adopt graph attention network to iteratively update representations of lung segments. To distinguish different infection degrees of left and right lungs, we further introduce a structural attention mechanism. Finally, we introduce demographic information and develop a multi-task learning framework to jointly perform both tasks of classification and regression. Experiments are conducted on a real dataset with 1687 chest CT scans, which includes 1328 mild cases and 359 severe cases. Experimental results show that our method achieves the best classification (e.g., 86.86% in terms of Area Under Curve) and regression (e.g., 0.58 in terms of Correlation Coefficient) performance, compared with other comparison methods.

[1]  Mohamed Abdel-Basset,et al.  MT-nCov-Net: A Multitask Deep-Learning Framework for Efficient Diagnosis of COVID-19 Using Tomography Scans , 2021, IEEE Transactions on Cybernetics.

[2]  F. Zhu,et al.  Machine Learning Algorithms are Superior to Conventional Regression Models in Predicting Risk Stratification of COVID-19 Patients , 2021, Risk management and healthcare policy.

[3]  J. Zhang,et al.  Pay attention to doctor–patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis , 2021, Information Fusion.

[4]  Lars Petersson,et al.  Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future , 2021, Sensors.

[5]  H. Lam,et al.  Classification of COVID-19 chest X-Ray and CT images using a type of dynamic CNN modification method , 2021, Computers in Biology and Medicine.

[6]  M. Lane-Fall,et al.  Development of an Anesthesiology Disaster Response Plan , 2021, Anesthesiology Clinics.

[7]  R. Sarkar,et al.  GraphCovidNet: A graph neural network based model for detecting COVID-19 from CT scans and X-rays of chest , 2021, Scientific Reports.

[8]  Shuihua Wang,et al.  ResGNet-C: A graph convolutional neural network for detection of COVID-19 , 2020, Neurocomputing.

[9]  Muhammet Fatih Aslan,et al.  CNN-based transfer learning–BiLSTM network: A novel approach for COVID-19 infection detection , 2020, Applied Soft Computing.

[10]  Mohammad Belayet Hossain,et al.  Attention-based VGG-16 model for COVID-19 chest X-ray image classification , 2020, Applied Intelligence.

[11]  Shadrokh Samavi,et al.  Bifurcated Autoencoder for Segmentation of COVID-19 Infected Regions in CT Images , 2020, ICPR Workshops.

[12]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[13]  Tsuyoshi Murata,et al.  Population Graph-Based Multi-Model Ensemble Method for Diagnosing Autism Spectrum Disorder , 2020, Sensors.

[14]  Su Ruan,et al.  Multi-task deep learning based CT imaging analysis for COVID-19 pneumonia: Classification and segmentation , 2020, Computers in Biology and Medicine.

[15]  Guorong Wu,et al.  Attention-Guided Deep Graph Neural Network for Longitudinal Alzheimer's Disease Analysis , 2020, MICCAI.

[16]  Lior Ness,et al.  Multi-task Learning for Detection and Classification of Cancer in Screening Mammography , 2020, MICCAI.

[17]  M. Sharkas,et al.  A multi-task pipeline with specialized streams for classification and segmentation of infection manifestations in COVID-19 scans , 2020, medRxiv.

[18]  Zhiyong Xu,et al.  A Noise-Robust Framework for Automatic Segmentation of COVID-19 Pneumonia Lesions From CT Images , 2020, IEEE Transactions on Medical Imaging.

[19]  S. Morozov,et al.  CT-Based COVID-19 triage: Deep multitask learning improves joint identification and severity quantification , 2020, Medical Image Analysis.

[20]  Cuntai Guan,et al.  Interpreting mechanisms of prediction for skin cancer diagnosis using multi-task learning , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  D. Shen,et al.  Joint prediction and time estimation of COVID-19 developing severe symptoms using chest CT scan , 2020, Medical Image Analysis.

[22]  Dijia Wu,et al.  Diagnosis of Coronavirus Disease 2019 (COVID-19) With Structured Latent Multi-View Representation Learning , 2020, IEEE Transactions on Medical Imaging.

[23]  G. Placidi,et al.  A light CNN for detecting COVID-19 from CT scans of the chest , 2020, Pattern Recognition Letters.

[24]  D.-P. Fan,et al.  Inf-Net: Automatic COVID-19 Lung Infection Segmentation From CT Images , 2020, IEEE Transactions on Medical Imaging.

[25]  Vijayan K. Asari,et al.  COVID_MTNet: COVID-19 Detection with Multi-Task Deep Learning Approaches , 2020, ArXiv.

[26]  Umut Ozkaya,et al.  Coronavirus (Covid-19) Classification Using CT Images by Machine Learning Methods , 2020, RTA-CSIT.

[27]  D. Shen,et al.  Lung Infection Quantification of COVID-19 in CT Images with Deep Learning , 2020, ArXiv.

[28]  Yan Zhao,et al.  Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus-Infected Pneumonia in Wuhan, China. , 2020, JAMA.

[29]  Z. Fayad,et al.  CT Imaging Features of 2019 Novel Coronavirus (2019-nCoV) , 2020, Radiology.

[30]  Yuan Zong,et al.  Sparse Graphic Attention LSTM for EEG Emotion Recognition , 2019, ICONIP.

[31]  Marleen de Bruijne,et al.  A joint 3D UNet-Graph Neural Network-based method for Airway Segmentation from chest CTs , 2019, MLMI@MICCAI.

[32]  James C. Gee,et al.  Interpretable Multimodality Embedding Of Cerebral Cortex Using Attention Graph Network For Identifying Bipolar Disorder , 2019, bioRxiv.

[33]  Dinggang Shen,et al.  Joint Classification and Regression via Deep Multi-Task Multi-Channel Learning for Alzheimer's Disease Diagnosis , 2019, IEEE Transactions on Biomedical Engineering.

[34]  Shadi Albarqouni,et al.  InceptionGCN: Receptive Field Aware Graph Convolutional Network for Disease Prediction , 2019, IPMI.

[35]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[36]  Ben Glocker,et al.  Disease prediction using graph convolutional networks: Application to Autism Spectrum Disorder and Alzheimer's disease , 2018, Medical Image Anal..

[37]  Yixin Chen,et al.  An End-to-End Deep Learning Architecture for Graph Classification , 2018, AAAI.

[38]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[39]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[40]  Sebastian Ruder,et al.  An Overview of Multi-Task Learning in Deep Neural Networks , 2017, ArXiv.

[41]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[42]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[43]  Roberto Cipolla,et al.  Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[44]  Ben Glocker,et al.  Distance Metric Learning Using Graph Convolutional Networks: Application to Functional Brain Networks , 2017, MICCAI.

[45]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[47]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[49]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[50]  Daoqiang Zhang,et al.  Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer's disease , 2012, NeuroImage.

[51]  Jing Li,et al.  Judging Correlation from Scatterplots and Parallel Coordinate Plots , 2010, Inf. Vis..

[52]  S. Sathiya Keerthi,et al.  Developing parallel sequential minimal optimization for fast training support vector machine , 2006, Neurocomputing.

[53]  Robert P. Sheridan,et al.  Random Forest: A Classification and Regression Tool for Compound Classification and QSAR Modeling , 2003, J. Chem. Inf. Comput. Sci..

[54]  Michael I. Jordan,et al.  On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes , 2001, NIPS.

[55]  M. W Gardner,et al.  Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciences , 1998 .

[56]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[57]  Tanja Hueber,et al.  Gaussian Processes For Machine Learning , 2016 .

[58]  Laurens van der Maaten,et al.  Accelerating t-SNE using tree-based algorithms , 2014, J. Mach. Learn. Res..

[59]  Joseph F. Tomashefski,et al.  Anatomy and Histology of the Lung , 2008 .

[60]  H. Martens,et al.  Modified Jack-knife estimation of parameter uncertainty in bilinear modelling by partial least squares regression (PLSR) , 2000 .