Customization of a Severity of Illness Score Using Local Electronic Medical Record Data

Purpose: Severity of illness (SOI) scores are traditionally based on archival data collected from a wide range of clinical settings. Mortality prediction using SOI scores tends to underperform when applied to contemporary cases or those that differ from the case-mix of the original derivation cohorts. We investigated the use of local clinical data captured from hospital electronic medical records (EMRs) to improve the predictive performance of traditional severity of illness scoring. Methods: We conducted a retrospective analysis using data from the Multiparameter Intelligent Monitoring in Intensive Care II (MIMIC-II) database, which contains clinical data from the Beth Israel Deaconess Medical Center in Boston, Massachusetts. A total of 17 490 intensive care unit (ICU) admissions with complete data were included, from 4 different service types: medical ICU, surgical ICU, coronary care unit, and cardiac surgery recovery unit. We developed customized SOI scores trained on data from each service type, using the clinical variables employed in the Simplified Acute Physiology Score (SAPS). In-hospital, 30-day, and 2-year mortality predictions were compared with those obtained from using the original SAPS using the area under the receiver–operating characteristics curve (AUROC) as well as the area under the precision-recall curve (AUPRC). Test performance in different cohorts stratified by severity of organ injury was also evaluated. Results: Most customized scores (30 of 39) significantly outperformed SAPS with respect to both AUROC and AUPRC. Enhancements over SAPS were greatest for patients undergoing cardiovascular surgery and for prediction of 2-year mortality. Conclusions: Custom models based on ICU-specific data provided better mortality prediction than traditional SAPS scoring using the same predictor variables. Our local data approach demonstrates the value of electronic data capture in the ICU, of secondary uses of EMR data, and of local customization of SOI scoring.