Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission: a prospective multicentre clinical outcomes evaluation of real-world patient data from US hospitals

Objective To validate performance of a machine learning algorithm for severe sepsis determination up to 48 hours before onset, and to evaluate the effect of the algorithm on in-hospital mortality, hospital length of stay, and 30-day readmission. Setting This cohort study includes a combined retrospective analysis and clinical outcomes evaluation: a dataset containing 510,497 patient encounters from 461 United States health centers for retrospective analysis, and a multiyear, multicenter clinical data set of real-world data containing 75,147 patient encounters from nine hospitals for clinical outcomes evaluation. Participants For retrospective analysis, 270,438 adult patients with at least one documented measurement of five out of six vital sign measurements were included. For clinical outcomes analysis, 17,758 adult patients who met two or more Systemic Inflammatory Response Syndrome (SIRS) criteria at any point during their stay were included. Results At severe sepsis onset, the MLA demonstrated an AUROC of 0.91 (95% CI 0.90, 0.92), which exceeded those of MEWS (0.71, P<001), SOFA (0.74; P<.001), and SIRS (0.62; P<.001). For severe sepsis prediction 48 hours in advance of onset, the MLA achieved an AUROC of 0.77 (95% CI 0.73, 0.80). For the clinical outcomes study, when using the MLA, hospitals saw an average 39.5% reduction of in-hospital mortality, a 32.3% reduction in hospital length of stay, and a 22.7% reduction in 30-day readmission rate. Conclusions The MLA accurately predicts severe sepsis onset up to 48 hours in advance using only readily available vital signs in retrospective validation. Reductions of in-hospital mortality, hospital length of stay, and 30-day readmissions were observed in real-world clinical use of the MLA. Results suggest this system may improve severe sepsis detection and patient outcomes over the use of rules-based sepsis detection systems. KEY POINTS Question Is a machine learning algorithm capable of accurate severe sepsis prediction, and does its clinical implementation improve patient mortality rates, hospital length of stay, and 30-day readmission rates? Findings In a retrospective analysis that included datasets containing a total of 585,644 patient encounters from 461 hospitals, the machine learning algorithm demonstrated an AUROC of 0.93 at time of severe sepsis onset, which exceeded those of MEWS (0.71), SOFA (0.74), and SIRS (0.62); and an AUROC of 0.77 for severe sepsis prediction 48 hours in advance of onset. In an analysis of real-world data from nine hospitals across 75,147 patient encounters, use of the machine learning algorithm was associated with a 39.5% reduction in in-hospital mortality, a 32.3% reduction in hospital length of stay, and a 22.7% reduction in 30-day readmission rate. Meaning The accurate and predictive nature of this algorithm may encourage early recognition of patients trending toward severe sepsis, and therefore improve sepsis related outcomes. STRENGTHS AND LIMITATIONS OF THIS STUDY A retrospective study of machine learning severe sepsis prediction from a dataset with 510,497 patient encounters demonstrates high accuracy up to 48 hours prior to onset. A multicenter clinical study of real-world data using this machine learning algorithm for severe sepsis alerts achieved reductions of in-hospital mortality, length of stay, and 30-day readmissions. The required presence of an ICD-9 code to classify a patient as severely septic in our retrospective analysis potentially limits our ability to accurately classify all patients. Only adults in US hospitals were included in this study. For the real-world section of the study, we cannot eliminate the possibility that implementation of a sepsis algorithm raised general awareness of sepsis within a hospital, which may lead to higher recognition of septic patients, independent of algorithm performance.

[1]  L. Levin,et al.  Biodiversity on the Rocks: Macrofauna Inhabiting Authigenic Carbonate at Costa Rica Methane Seeps , 2015, PloS one.

[2]  M. Levy,et al.  Empiric Antibiotic Treatment Reduces Mortality in Severe Sepsis and Septic Shock From the First Hour: Results From a Guideline-Based Performance Improvement Program* , 2014, Critical care medicine.

[3]  Susan Gruber,et al.  Incidence and Trends of Sepsis in US Hospitals Using Clinical vs Claims Data, 2009-2014 , 2017, JAMA.

[4]  M. Cvach.  Monitor alarm fatigue: an integrative review. , 2012, Biomedical instrumentation & technology.

[5]  Shamim Nemati,et al.  Multiscale network representation of physiological time series for early prediction of sepsis , 2017, Physiological measurement.

[6]  Steven Horng,et al.  Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning , 2017, PloS one.

[7]  Christopher W. Barton,et al.  A computational approach to early sepsis detection , 2016, Comput. Biol. Medicine.

[8]  Mitchell M. Levy,et al.  2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference , 2003, Intensive Care Medicine.

[9]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[10]  R. Bellomo,et al.  The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). , 2016, JAMA.

[11]  Uli K. Chettipally,et al.  Multicenter validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU , 2018, bioRxiv.

[12]  T. Buchman,et al.  Filtering authentic sepsis arising in the ICU using administrative codes coupled to a SIRS screening protocol , 2017, Journal of critical care.

[13]  M. Levy,et al.  2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference , 2003, Intensive care medicine.

[14]  A. Lynn-Palevsky,et al.  Evaluating a sepsis prediction machine learning algorithm using minimal electronic health record data in the emergency department and intensive care unit , 2017, bioRxiv.

[15]  E. Adrario,et al.  Effect of Performance Improvement Programs on Compliance with Sepsis Bundles and Mortality: A Systematic Review and Meta-Analysis of Observational Studies , 2015, PloS one.

[16]  F. Moore,et al.  Early Diagnosis and Evidence-Based Care of Surgical Sepsis , 2013, Journal of intensive care medicine.

[17]  G. Clermont,et al.  Epidemiology of severe sepsis in the United States: Analysis of incidence, outcome, and associated costs of care , 2001, Critical care medicine.

[18]  Fred H. Hamker,et al.  Septic Shock Diagnosis by Neural Networks and Rule Based Systems , 2002 .

[19]  Shinichiro Kurosawa,et al.  Sepsis: multiple abnormalities, heterogeneous responses, and evolving understanding. , 2013, Physiological reviews.

[20]  Ritankar Das,et al.  Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial , 2017, BMJ Open Respiratory Research.

[21]  C. Torio,et al.  National Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2011 , 2013 .

[22]  Theodore J Iwashyna,et al.  Identifying Patients With Severe Sepsis Using Administrative Claims: Patient-Level Validation of the Angus Implementation of the International Consensus Conference Definition of Severe Sepsis , 2014, Medical care.

[23]  Joshua A. Doherty,et al.  Early prediction of septic shock in hospitalized patients. , 2010, Journal of hospital medicine.

[24]  Fred L. Drake,et al.  The Python Language Reference Manual , 1999 .

[25]  Uli K. Chettipally,et al.  Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU , 2018, BMJ Open.

[26]  R. Sorrentino.  Large standard deviations and logarithmic-normality , 2010, Fly.

[27]  Ritankar Das,et al.  Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units , 2017, BMJ open quality.

[28]  Christopher W Seymour,et al.  Developing a New Definition and Assessing New Clinical Criteria for Septic Shock: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). , 2016, JAMA.

[29]  Hien Nguyen,et al.  From vital signs to clinical outcomes for patients with sepsis: a machine learning basis for a clinical decision support system , 2014, J. Am. Medical Informatics Assoc..

[30]  Uli K. Chettipally,et al.  Prediction of Sepsis in the Intensive Care Unit With Minimal Electronic Health Record Data: A Machine Learning Approach , 2016, JMIR medical informatics.

[31]  Hamid Mohamadlou,et al.  High-performance detection and early prediction of septic shock for alcohol-use disorder patients , 2016, Annals of medicine and surgery.

[32]  Renda Soylemez Wiener,et al.  Two Decades of Mortality Trends Among Patients With Severe Sepsis: A Comparative Meta-Analysis* , 2014, Critical care medicine.

[33]  Eiji Kajii,et al.  Importance of vital signs to the early diagnosis and severity of sepsis: association between vital signs and sequential organ failure assessment score in patients with sepsis. , 2012, Internal medicine.