An artificial intelligence system for predicting the deterioration of COVID-19 patients in the emergency department

During the COVID-19 pandemic, rapid and accurate triage of patients at the emergency department is critical to inform decision-making. We propose a data-driven approach for automatic prediction of deterioration risk using a deep neural network that learns from chest X-ray images, and a gradient boosting model that learns from routine clinical variables. Our AI prognosis system, trained using data from 3,661 patients, achieves an AUC of 0.786 (95% CI: 0.742-0.827) when predicting deterioration within 96 hours. The deep neural network extracts informative areas of chest X-ray images to assist clinicians in interpreting the predictions, and performs comparably to two radiologists in a reader study. In order to verify performance in a real clinical setting, we silently deployed a preliminary version of the deep neural network at NYU Langone Health during the first wave of the pandemic, which produced accurate predictions in real-time. In summary, our findings demonstrate the potential of the proposed system for assisting front-line physicians in the triage of COVID-19 patients.

[1]  Kyunghyun Cho,et al.  Classifier-agnostic saliency map extraction , 2018, AAAI.

[2]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[3]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[4]  Kyunghyun Cho,et al.  High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks , 2017, ArXiv.

[5]  Nan Wu,et al.  An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization , 2020, Medical Image Anal..

[6]  Balasubramanian Narasimhan,et al.  A scalable discrete-time survival model for neural networks , 2018, PeerJ.

[7]  Ronald M. Summers,et al.  ChestX-ray: Hospital-Scale Chest X-ray Database and Benchmarks on Weakly Supervised Classification and Localization of Common Thorax Diseases , 2019, Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics.

[8]  W. Liang,et al.  Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed Tomography , 2020, Cell.

[9]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[10]  Kyunghyun Cho,et al.  Globally-Aware Multiple Instance Classifier for Breast Cancer Screening , 2019, MLMI@MICCAI.

[11]  R. Maroldi,et al.  COVID-19 outbreak in Italy: experimental chest X-ray scoring system for quantifying and monitoring disease progression , 2020, La radiologia medica.

[12]  Eun Ji Kim,et al.  Machine learning to assist clinical decision-making during the COVID-19 pandemic , 2020, Bioelectronic Medicine.

[13]  Been Kim,et al.  Sanity Checks for Saliency Maps , 2018, NeurIPS.

[14]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[15]  David A. Clifton,et al.  Machine Learning for Clinical Outcome Prediction , 2020, IEEE Reviews in Biomedical Engineering.

[16]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[17]  Asif Iqbal Khan,et al.  CoroNet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images , 2020, Computer Methods and Programs in Biomedicine.

[18]  Deniz Korkmaz,et al.  COVIDiagnosis-Net: Deep Bayes-SqueezeNet based diagnosis of the coronavirus disease 2019 (COVID-19) from X-ray images , 2020, Medical Hypotheses.

[19]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Vaishali,et al.  Classification of COVID-19 patients from chest CT images using multi-objective differential evolution–based convolutional neural networks , 2020, European Journal of Clinical Microbiology & Infectious Diseases.

[21]  Charles E Kahn,et al.  How Might AI and Chest Imaging Help Unravel COVID-19’s Mysteries? , 2020, Radiology. Artificial intelligence.

[22]  G. Heinze,et al.  Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal , 2020, BMJ.

[23]  Maia Dorsett,et al.  Point of no return: COVID-19 and the U.S. healthcare system: An emergency physician’s perspective , 2020, Science Advances.

[24]  R. C. Macridis.  A review , 1963 .

[25]  Ali Narin,et al.  Automatic detection of coronavirus disease (COVID-19) using X-ray images and deep convolutional neural networks , 2020, Pattern Analysis and Applications.

[26]  Michael Gao,et al.  Prospective and External Evaluation of a Machine Learning Model to Predict In-Hospital Mortality of Adults at Time of Admission. , 2020, JAMA network open.

[27]  Uri Shaham,et al.  DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network , 2016, BMC Medical Research Methodology.

[28]  Andrew Y. Ng,et al.  CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning , 2017, ArXiv.

[29]  Thomas J Oxley,et al.  Large-Vessel Stroke as a Presenting Feature of Covid-19 in the Young , 2020, The New England journal of medicine.

[30]  Lei Wang,et al.  SDFN: Segmentation-based Deep Fusion Network for Thoracic Disease Classification in Chest X-ray Images , 2018, Comput. Medical Imaging Graph..

[31]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[32]  K. Cao,et al.  Using Artificial Intelligence to Detect COVID-19 and Community-acquired Pneumonia Based on Pulmonary CT: Evaluation of the Diagnostic Accuracy , 2020 .

[33]  Peter J McKenna,et al.  Emergency department and hospital crowding: causes, consequences, and cures , 2019, Clinical and experimental emergency medicine.

[34]  Nils Kucher,et al.  Venous and arterial thromboembolic complications in COVID-19 patients admitted to an academic hospital in Milan, Italy , 2020, Thrombosis Research.

[35]  Xun Zhu,et al.  Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data , 2018, PLoS Comput. Biol..

[36]  Ali Pourmand,et al.  Redesigning emergency department operations amidst a viral pandemic , 2020, The American Journal of Emergency Medicine.

[37]  Simone Busoni,et al.  Chest X-ray in new Coronavirus Disease 2019 (COVID-19) infection: findings and correlation with clinical outcome , 2020, La radiologia medica.

[38]  Yi-Zeng Liang,et al.  Monte Carlo cross validation , 2001 .

[39]  Imane Allaouzi,et al.  A Novel Approach for Multi-Label Chest X-Ray Classification of Common Thorax Diseases , 2019, IEEE Access.

[40]  Yan Bai,et al.  A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis , 2020, European Respiratory Journal.

[41]  Paul H. Yi,et al.  Generalizability of Deep Learning Tuberculosis Classifier to COVID-19 Chest Radiographs , 2020, Journal of thoracic imaging.

[42]  K. Cao,et al.  Artificial Intelligence Distinguishes COVID-19 from Community Acquired Pneumonia on Chest CT , 2020, Radiology.

[43]  Matthew A. Warner,et al.  Stop Doing Needless Things! Saving Healthcare Resources During COVID-19 and Beyond , 2020, Journal of General Internal Medicine.

[44]  Kai Liu,et al.  Clinical features of COVID-19 in elderly patients: A comparison with young and middle-aged patients , 2020, Journal of Infection.

[45]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[46]  Antonella Santone,et al.  Explainable Deep Learning for Pulmonary Disease and Coronavirus COVID-19 Detection from X-rays , 2020, Computer Methods and Programs in Biomedicine.

[47]  Ronald M. Summers,et al.  ChestX-ray: Hospital-Scale Chest X-ray Database and Benchmarks on Weakly Supervised Classification and Localization of Common Thorax Diseases , 2019, Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics.

[48]  Russell M Viner,et al.  Kawasaki-like disease: emerging complication during the COVID-19 pandemic , 2020, The Lancet.

[49]  Mark Finkelstein,et al.  Clinical and Chest Radiography Features Determine Patient Outcomes in Young and Middle-aged Adults with COVID-19 , 2020, Radiology.

[50]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[51]  Tie-Yan Liu,et al.  A Highly Efficient Gradient Boosting Decision Tree , 2017, NIPS 2017.

[52]  M. Kuo,et al.  Frequency and Distribution of Chest Radiographic Findings in COVID-19 Positive Patients , 2019, Radiology.

[53]  Tanaka.  The role of , 2000, Journal of insect physiology.

[54]  Ali S. Raja,et al.  Creating a COVID-19 surge clinic to offload the emergency department , 2020, The American Journal of Emergency Medicine.

[55]  Jianhua Yao,et al.  Early triage of critically ill COVID-19 patients using deep learning , 2020, Nature Communications.

[56]  Yaping Huang,et al.  Multi-label chest X-ray image classification via category-wise residual attention learning , 2020, Pattern Recognit. Lett..

[57]  Roberto Maroldi,et al.  COVID-19 outbreak in Italy: experimental chest X-ray scoring system for quantifying and monitoring disease progression , 2020, La radiologia medica.

[58]  H. Kauczor,et al.  The Role of Chest Imaging in Patient Management during the COVID-19 Pandemic: A Multinational Consensus Statement from the Fleischner Society , 2020, Radiology.

[59]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[60]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[61]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  P. Singh,et al.  Automated assessment of COVID-19 pulmonary disease severity on chest radiographs using convolutional Siamese neural networks , 2020, medRxiv.

[63]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[64]  Kyunghyun Cho,et al.  Classifier-agnostic saliency map extraction , 2020, Comput. Vis. Image Underst..

[65]  Max Welling,et al.  Attention-based Deep Multiple Instance Learning , 2018, ICML.

[66]  Weirong Li,et al.  Exploring the active mechanism of berberine against HCC by systematic pharmacology and experimental validation , 2019, Molecular medicine reports.

[67]  Tie-Yan Liu,et al.  LightGBM: A Highly Efficient Gradient Boosting Decision Tree , 2017, NIPS.

[68]  Joao M. C. Sousa,et al.  Clinical Decision Support Systems for Triage in the Emergency Department using Intelligent Systems: a Review , 2020, Artif. Intell. Medicine.

[69]  Farah E. Shamout,et al.  Deep Interpretable Early Warning System for the Detection of Clinical Deterioration , 2020, IEEE Journal of Biomedical and Health Informatics.

[70]  Stefan Seebacher,et al.  Challenges in the Deployment and Operation of Machine Learning in Practice , 2019, ECIS.

[71]  U. Rajendra Acharya,et al.  Automated detection of COVID-19 cases using deep neural networks with X-ray images , 2020, Computers in Biology and Medicine.

[72]  Ankur Teredesai,et al.  Interpretable Machine Learning in Healthcare , 2018, 2018 IEEE International Conference on Healthcare Informatics (ICHI).

[73]  Richard D Riley,et al.  Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal , 2020 .