Early detection of sepsis utilizing deep learning on electronic health record event sequences

BACKGROUND The timeliness of detection of a sepsis incidence in progress is a crucial factor in the outcome for the patient. Machine learning models built from data in electronic health records can be used as an effective tool for improving this timeliness, but so far, the potential for clinical implementations has been largely limited to studies in intensive care units. This study will employ a richer data set that will expand the applicability of these models beyond intensive care units. Furthermore, we will circumvent several important limitations that have been found in the literature: (1) Model evaluations neglect the clinical consequences of a decision to start, or not start, an intervention for sepsis. (2) Models are evaluated shortly before sepsis onset without considering interventions already initiated. (3) Machine learning models are built on a restricted set of clinical parameters, which are not necessarily measured in all departments. (4) Model performance is limited by current knowledge of sepsis, as feature interactions and time dependencies are hard-coded into the model. METHODS In this study, we present a model to overcome these shortcomings using a deep learning approach on a diverse multicenter data set. We used retrospective data from multiple Danish hospitals over a seven-year period. Our sepsis detection system is constructed as a combination of a convolutional neural network and a long short-term memory network. We assess model quality by standard concepts of accuracy as well as clinical usefulness, and we suggest a retrospective assessment of interventions by looking at intravenous antibiotics and blood cultures preceding the prediction time. RESULTS Results show performance ranging from AUROC 0.856 (3 h before sepsis onset) to AUROC 0.756 (24 h before sepsis onset). Evaluating the clinical utility of the model, we find that a large proportion of septic patients did not receive antibiotic treatment or blood culture at the time of the sepsis prediction, and the model could, therefore, facilitate such interventions at an earlier point in time. CONCLUSION We present a deep learning system for early detection of sepsis that can learn characteristics of the key factors and interactions from the raw event sequence data itself, without relying on a labor-intensive feature extraction work. Our system outperforms baseline models, such as gradient boosting, which rely on specific data elements and therefore suffer from many missing values in our dataset.

[1]  J. Habbema,et al.  The measurement of performance in probabilistic diagnosis. II. Trustworthiness of the exact values of the diagnostic probabilities. , 1978, Methods of information in medicine.

[2]  C. Pedersen,et al.  The Danish Civil Registration System , 2011, Scandinavian journal of public health.

[3]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[4]  R. Bellomo,et al.  The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). , 2016, JAMA.

[5]  Alexander Binder,et al.  Explaining nonlinear classification decisions with deep Taylor decomposition , 2015, Pattern Recognit..

[6]  Jiaquan Xu,et al.  Deaths: final data for 2010. , 2013, National vital statistics reports : from the Centers for Disease Control and Prevention, National Center for Health Statistics, National Vital Statistics System.

[7]  Yvonne Vergouwe,et al.  Towards better clinical prediction models: seven steps for development and an ABCD for validation. , 2014, European heart journal.

[8]  Mitchell M. Levy,et al.  2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference , 2003, Intensive Care Medicine.

[9]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Parisa Rashidi,et al.  Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis , 2017, IEEE Journal of Biomedical and Health Informatics.

[11]  Harry J de Koning,et al.  Risk prediction models for selection of lung cancer screening candidates: A retrospective validation study , 2017, PLoS medicine.

[12]  Steven Horng,et al.  Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning , 2017, PloS one.

[13]  Tara N. Sainath,et al.  Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[14]  Christopher W. Barton,et al.  A computational approach to early sepsis detection , 2016, Comput. Biol. Medicine.

[15]  P. Bossuyt,et al.  An assessment of the relationship between clinical utility and predictive ability measures and the impact of mean risk in the population , 2014, BMC Medical Research Methodology.

[16]  Benjamin Recht,et al.  Do CIFAR-10 Classifiers Generalize to CIFAR-10? , 2018, ArXiv.

[17]  I. Kohane,et al.  Biases in electronic health record data due to processes within the healthcare system: retrospective observational study , 2018, British Medical Journal.

[18]  Valentin Rousson,et al.  Decision curve analysis revisited: overall net benefit, relationships to ROC curve analysis, and application to case-control studies , 2011, BMC Medical Informatics Decis. Mak..

[19]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[20]  Alan E. Jones,et al.  Lactate clearance vs central venous oxygen saturation as goals of early sepsis therapy: a randomized clinical trial. , 2010, JAMA.

[21]  G. Clermont,et al.  Epidemiology of severe sepsis in the United States: Analysis of incidence, outcome, and associated costs of care , 2001, Critical care medicine.

[22]  T. Rea,et al.  Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). , 2016, JAMA.

[23]  Ritankar Das,et al.  Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial , 2017, BMJ Open Respiratory Research.

[24]  Alexander Binder,et al.  Layer-Wise Relevance Propagation for Deep Neural Network Architectures , 2016 .

[25]  Terry Hainsworth,et al.  Improving the prevention of cardiovascular disease. , 2006, Nursing times.

[26]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[27]  N. Obuchowski,et al.  Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures , 2010, Epidemiology.

[28]  Katherine A. Heller,et al.  Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier , 2017, ICML.

[29]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[30]  Sigrun Alba Johannesdottir Schmidt,et al.  The Danish National Patient Registry: a review of content, data quality, and research potential , 2015, Clinical epidemiology.

[31]  Uli K. Chettipally,et al.  Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU , 2018, BMJ Open.

[32]  Chieh-Chen Wu,et al.  Prediction of sepsis patients using machine learning approach: A meta-analysis , 2019, Comput. Methods Programs Biomed..

[33]  E. Elkin,et al.  Decision Curve Analysis: A Novel Method for Evaluating Prediction Models , 2006, Medical decision making : an international journal of the Society for Medical Decision Making.

[34]  Shamim Nemati,et al.  An Interpretable Machine Learning Model for Accurate Prediction of Sepsis in the ICU , 2017, Critical care medicine.

[35]  Marcus A. Badgeley,et al.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study , 2018, PLoS medicine.

[36]  Rajesh Talluri,et al.  Using the weighted area under the net benefit curve for decision curve analysis , 2016, BMC Medical Informatics and Decision Making.

[37]  Steve Halligan,et al.  Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: A discussion and proposal for an alternative approach , 2015, European Radiology.

[38]  Katherine A. Heller,et al.  An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection , 2017, MLHC.

[40]  Anders Perner,et al.  Sygdomsbyrde og definitioner af sepsis hos voksne , 2018 .

[41]  J. Vincent,et al.  The Clinical Challenge of Sepsis Identification and Monitoring , 2016, PLoS medicine.

[42]  Yann LeCun,et al.  Very Deep Convolutional Networks for Text Classification , 2016, EACL.

[43]  Andrew J Vickers,et al.  Everything you always wanted to know about evaluating prediction models (but were too afraid to ask). , 2010, Urology.