Identifying poor-quality hospitals. Can hospital mortality rates detect quality problems for medical diagnoses?

OBJECTIVES Many groups involved in health care are very interested in using external quality indices, such as risk-adjusted mortality rates, to examine hospital quality. The authors evaluated the feasibility of using mortality rates for medical diagnoses to identify poor-quality hospitals. METHODS The Monte Carlo simulation model was used to examine whether mortality rates could distinguish 172 average-quality hospitals from 19 poor-quality hospitals (5% versus 25% of deaths being preventable, respectively), using the largest diagnosis-related groups (DRGs) for cardiac, gastrointestinal, cerebrovascular, and pulmonary diseases as well as an aggregate of all medical DRGs. Discharge counts and observed death rates for all 191 Michigan hospitals were obtained from the Michigan Inpatient Database. Positive predictive value (PPV), sensitivity, and area under the receiver operating characteristic curve were calculated for mortality outlier status as an indicator of poor-quality hospitals. Sensitivity analysis was performed under varying assumptions about the time period of evaluation, quality differences between hospitals, and unmeasured variability in hospital casemix. RESULTS For individual DRG groups, mortality rates were a poor measure of quality, even using the optimistic assumption of perfect casemix adjustment. For acute myocardial infarction, high mortality rate outlier status (using 2 years of data and a 0.05 probability cutoff) had a PPV of only 24%, thus, more than three fourths of those labeled poor-quality hospitals (high mortality rate outliers) actually would have average quality. If we aggregate all medical DRGs and continue to assume very large quality differences and perfect casemix adjustment, the sensitivity for detecting poor-quality hospitals is 35% and PPV is 52%. Even for this extreme case, the PPV is very sensitive to introduction of small amounts of unmeasured casemix differences between hospitals. CONCLUSION Although they may be useful for some surgical diagnoses, DRG-specific hospital mortality rates probably cannot accurately detect poor-quality outliers for medical diagnoses. Even collapsing to all medical DRGs, hospital mortality rates seem unlikely to be accurate predictors of poor quality, and punitive measures based on high mortality rates frequently would penalize good or average hospitals.

[1]  D Draper,et al.  Interpreting hospital mortality data. How can we proceed? , 1988, JAMA.

[2]  Frans Van de Werf,et al.  An international randomized trial comparing four thrombolytic strategies for acute myocardial infarction. , 1993, The New England journal of medicine.

[3]  M. Blumberg.  Biased estimates of expected acute myocardial infarction mortality using MedisGroups admission severity groups. , 1991, JAMA.

[4]  R H Brook,et al.  Watching the doctor-watchers. How well do peer review organization methods detect hospital care quality problems? , 1992, JAMA.

[5]  Bell Nn.  HEDIS (Health Employer Data and Information Set): are we finally comparing apples to apples? , 1994 .

[6]  E L Hannan,et al.  Percutaneous transluminal coronary angioplasty in New York State. Risk factors and outcomes. , 1992, JAMA.

[7]  A. Rimm,et al.  Assessing providers of coronary revascularization: a method for peer review organizations. , 1992, American journal of public health.

[8]  D Draper,et al.  Hospital characteristics and quality of care. , 1992, JAMA.

[9]  R. Elashoff,et al.  Flaws in mortality data. The hazards of ignoring comorbid disease. , 1988, JAMA.

[10]  M. Freeland,et al.  Selective contracting for hospital care based on volume, quality, and price: prospects, problems, and unanswered questions. , 1987, Journal of health politics, policy and law.

[11]  A. Rimm,et al.  The relationship between adjusted hospital mortality and the results of peer review. , 1993, Health services research.

[12]  R H Brook,et al.  Explaining variations in hospital death rates. Randomness, severity of illness, quality of care. , 1990, JAMA.

[13]  K. Lohr,et al.  Differences among hospitals in Medicare patient mortality. , 1989, Health services research.

[14]  E. Hannan,et al.  Adult open heart surgery in New York State. An analysis of risk factors and hospital mortality rates. , 1990, JAMA.

[15]  G. Coffman,et al.  Predicting In-Hospital Mortality: A Comparison of Severity Measurement Approaches , 1992, Medical care.

[16]  T A Brennan,et al.  Reliability and Validity of Judgments Concerning Adverse Events Suffered by Hospitalized Patients , 1989, Medical care.

[17]  E. Veys,et al.  Letter: HL-A and infective sacroileitis. , 1974, Lancet.

[18]  J. Ware,et al.  Involving consumers in quality of care assessment. , 1988, Health affairs.

[19]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[20]  A M Bernard,et al.  Evaluating the Care of General Medicine Inpatients: How Good Is Implicit Review? , 1993, Annals of Internal Medicine.

[21]  R. Brook,et al.  The effect of alternative case-mix adjustments on mortality differences between municipal and voluntary hospitals in New York City. , 1994, Health services research.

[22]  D Draper,et al.  Interpreting hospital mortality data. The role of clinical risk adjustment. , 1988, JAMA.

[23]  M. Radford,et al.  Process and Outcome of Care for Acute Myocardial Infarction among Medicare Beneficiaries in Connecticut: A Quality Improvement Demonstration Project , 1995, Annals of Internal Medicine.

[24]  P Sharkey,et al.  The importance of severity of illness in assessing hospital mortality. , 1990, JAMA.

[25]  J. S. Hixson,et al.  Efficacy of Statistical Outlier Analysis for Monitoring Quality of Care , 1991 .

[26]  A A Rimm,et al.  Hospital characteristics and mortality rates. , 1989, The New England journal of medicine.

[27]  E. Hannan,et al.  The decline in coronary artery bypass graft surgery mortality in New York State. The role of surgeon volume. , 1995, JAMA.

[28]  R H Brook,et al.  Preventable deaths: who, how often, and why? , 1988, Annals of internal medicine.

[29]  R. C. Bradbury,et al.  Interhospital variations in admission severity-adjusted hospital mortality and morbidity. , 1991, Health services research.

[30]  E L Hannan,et al.  Clinical Versus Administrative Data Bases for CABG Surgery: Does it Matter , 1992, Medical care.

[31]  Robert H. Brook,et al.  Hospital Inpatient Mortality , 1987 .

[32]  P. Sanazaro,et al.  A critique of the use of generic screening in quality assessment. , 1991, JAMA.

[33]  L. McMahon,et al.  Socioeconomic Influence on Small Area Hospital Utilization , 1993, Medical care.

[34]  D Draper,et al.  Predicting hospital-associated mortality for Medicare patients. A method for patients with stroke, pneumonia, acute myocardial infarction, and congestive heart failure. , 1988, JAMA.

[35]  L. McMahon,et al.  An Evaluation of Generic Screens for Poor Quality of Hospital Care on a General Medicine Service , 1993, Medical care.

[36]  R H Brook,et al.  Adjusted hospital death rates: a potential screen for quality of medical care. , 1987, American journal of public health.

[37]  M. Pine,et al.  Potential Effectiveness of Comparative Casemis-corrected Adverse Outcome Rates as Quality Monitors in a Simulated Hospital System , 1989, Medical decision making : an international journal of the Society for Medical Decision Making.

[38]  M. Pine,et al.  Potential Effectiueness of Quality Assurance Screening Using Large but Imperfect Databases , 1990, Medical decision making : an international journal of the Society for Medical Decision Making.

[39]  L. McMahon,et al.  Measuring outcomes of hospital care using multiple risk-adjusted indexes. , 1991, Health services research.