Performance of risk-adjusted control charts to monitor in-hospital mortality of intensive care unit patients: A simulation study*

Objectives:Increases in case-mix adjusted mortality may be indications of decreasing quality of care. Risk-adjusted control charts can be used for in-hospital mortality monitoring in intensive care units by issuing a warning signal when there are more deaths than expected. The aim of this study was to systematically assess and compare, by computer simulation, expected delay before a warning signal was given for an upward shift in mortality rate in intensive care mortality data by different risk-adjusted control charts. Design:We compared the efficiency of the risk-adjusted P-chart, risk-adjusted Additive P-chart, risk-adjusted Multiplicative P-chart, monthly Standardized Mortality Ratio, risk-adjusted Cumulative Sum, risk-adjusted Resetting Sequential Probability Ratio Test, and risk-adjusted Exponentially Weighted Moving Average control chart to detect an upward shift in mortality rate in eight different scenarios that varied by mortality increase factor and monthly patient volume. Setting:Adult intensive care units in The Netherlands. Patients:Patients admitted to 73 intensive care units from the Dutch National Intensive Care Evaluation quality registry from the year 2009. Interventions:None. Measurements:We compared the performance of the different risk-adjusted control charts by the median time-to-signal and the 6-month detection rate. Main results:In all eight scenarios, the risk-adjusted Exponentially Weighted Moving Average control chart had the shortest median time-to-signal, and in four, the highest 6-month detection rate. The median time-to-signal for an average volume intensive care unit (i.e., 50 admissions per month) with an increase in mortality rate of R = 1.50 on the odds scale was 9 months for the risk-adjusted Exponentially Weighted Moving Average control chart. Conclusions:The risk-adjusted Exponentially Weighted Moving Average control chart signaled the fastest in most of the simulated scenarios and is therefore superior in detecting increases in in-hospital mortality of intensive care patients compared to the other types of risk-adjusted control charts.

[1]  R. G. Cornell.  Statistical Methods in Cancer Research, Volume II: The Design and Analysis of Cohort Studies , 1992 .

[2]  Eric L. Eisenstein,et al.  Risk-adjusted control charts for health care assessment , 1996, Ann. Oper. Res..

[3]  D. Hosmer,et al.  A comparison of goodness-of-fit tests for the logistic regression model. , 1997, Statistics in medicine.

[4]  M. R. Reynolds,et al.  The SPRT chart for monitoring a proportion , 1998 .

[5]  S. Steiner,et al.  Monitoring surgical performance using risk-adjusted cumulative sum charts. , 2000, Biostatistics.

[6]  R. Carey.  How do you know that your care is improving? Part II: Using control charts to learn from your data. , 2002, The Journal of ambulatory care management.

[7]  N. Best,et al.  Following Shipman: a pilot system for monitoring mortality rates in primary care , 2003, The Lancet.

[8]  R. F. Hart,et al.  Application of Attribute Control Charts to Risk‐Adjusted Data for Monitoring and Improving Health Care Performance , 2003, Quality management in health care.

[9]  V T Farewell,et al.  Use of risk-adjusted CUSUM and RSPRTcharts for monitoring in medical contexts , 2003, Statistical methods in medical research.

[10]  David A Cook,et al.  The development of risk adjusted control charts and machine learning models to monitor the mortality rate of intensive care unit patients , 2004 .

[11]  Vern T. Farewell,et al.  An overview of risk‐adjusted charts , 2004 .

[12]  J. Zimmerman,et al.  Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality assessment for today’s critically ill patients* , 2006, Critical care medicine.

[13]  David A Cook,et al.  Process monitoring in intensive care with the use of cumulative expected minus observed mortality and risk-adjusted p charts , 2006, Critical care.

[14]  Peter A. Rogerson,et al.  Formulas for the Design of CUSUM Quality Control Charts , 2006 .

[15]  Patrick Royston,et al.  The design of simulation studies in medical statistics , 2006, Statistics in medicine.

[16]  Mats Brommels,et al.  Application of statistical process control in healthcare improvement: systematic review , 2007, Quality and Safety in Health Care.

[17]  B. Guthrie,et al.  Routine mortality monitoring for detecting mass murder in UK general practice: test of effectiveness using modelling. , 2008, The British journal of general practice : the journal of the Royal College of General Practitioners.

[18]  Graeme K Hart,et al.  Review of the application of risk-adjusted charts to analyse mortality outcomes in critical care. , 2008, Critical care and resuscitation : journal of the Australasian Academy of Critical Care Medicine.

[19]  F. Merletti,et al.  Monitoring the performance of intensive care units using the variable life-adjusted display: a simulation study to explore its applicability and efficiency. , 2009, Journal of evaluation in clinical practice.

[20]  Ameen Abu-Hanna,et al.  Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment , 2013 .

[21]  M. Coory,et al.  Exponentially weighted moving average charts to compare observed and expected values for monitoring risk-adjusted hospital indicators , 2011, Quality and Safety in Health Care.