Comparison of “Risk-Adjusted” Hospital Outcomes

Background— A frequent challenge in outcomes research is the comparison of rates from different populations. One common example with substantial health policy implications involves the determination and comparison of hospital outcomes. The concept of “risk-adjusted” outcomes is frequently misunderstood, particularly when it is used to justify the direct comparison of performance at 2 specific institutions. Methods and Results— Data from 14 Massachusetts hospitals were analyzed for 4393 adults undergoing isolated coronary artery bypass graft surgery in 2003. Mortality estimates were adjusted using clinical data prospectively collected by hospital personnel and submitted to a data coordinating center designated by the state. The primary outcome was hospital-specific, risk-standardized, 30-day all-cause mortality after surgery. Propensity scores were used to assess the comparability of case mix (covariate balance) for each Massachusetts hospital relative to the pool of patients undergoing coronary artery bypass grafting surgery at the remaining hospitals and for selected pairwise comparisons. Using hierarchical logistic regression, we indirectly standardized the mortality rate of each hospital using its expected rate. Predictive cross-validation was used to avoid underidentification of true outlying hospitals. Overall, there was sufficient overlap between the case mix of each hospital and that of all other Massachusetts hospitals to justify comparison of individual hospital performance with that of the remaining hospitals. As expected, some pairwise hospital comparisons indicated lack of comparability. This finding illustrates the fallacy of assuming that risk adjustment per se is sufficient to permit direct side-by-side comparison of healthcare providers. In some instances, such analyses may be facilitated by the use of propensity scores to improve covariate balance between institutions and to justify such comparisons. Conclusions— Risk-adjusted outcomes, commonly the focus of public report cards, have a specific interpretation. Using indirect standardization, these outcomes reflect a provider's performance for its specific case mix relative to the expected performance of an average provider for that same case mix. Unless study design or post hoc adjustments have resulted in reasonable overlap of case-mix distributions, such risk-adjusted outcomes should not be used to directly compare one institution with another.

[1]  S. Normand,et al.  An Administrative Claims Model Suitable for Profiling Hospital Performance Based on 30-Day Mortality Rates Among Patients With an Acute Myocardial Infarction , 2006, Circulation.

[2]  R. D'Agostino.  Propensity Scores in Cardiovascular Research , 2007, Circulation.

[3]  Harlan M Krumholz,et al.  An Administrative Claims Model Suitable for Profiling Hospital Performance Based on 30-Day Mortality Rates Among Patients With Heart Failure , 2006, Circulation.

[4]  D. Rubin.  Causal Inference Using Potential Outcomes , 2005 .

[5]  Keying Ye,et al.  Applied Bayesian Modeling and Causal Inference From Incomplete-Data Perspectives , 2005, Technometrics.

[6]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[7]  P. Holland.  Statistics and Causal Inference , 1985 .

[8]  L. Iezzoni.  Risk Adjustment for Measuring Healthcare Outcomes , 1994 .

[9]  Sharon-Lise T Normand,et al.  On the use of discrete choice models for causal inference , 2005, Statistics in medicine.

[10]  B J McNeil,et al.  Current issues in profiling quality of care. , 1992, Inquiry : a journal of medical care organization, provision and financing.

[11]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[12]  D. Rubin,et al.  Constructing a Control Group Using Multivariate Matched Sampling Methods That Incorporate the Propensity Score , 1985 .

[13]  D. Rubin,et al.  Reducing Bias in Observational Studies Using Subclassification on the Propensity Score , 1984 .

[14]  D. Rubin.  The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials , 2007, Statistics in medicine.

[15]  D. Rubin,et al.  On the Nature and Discovery of Structure: Comment , 1984 .

[16]  Andrew Gelman,et al.  Applied Bayesian Modeling And Causal Inference From Incomplete-Data Perspectives , 2005 .

[17]  D. Rubin,et al.  Causal Inference in Retrospective Studies , 1987 .

[18]  J. Fleiss,et al.  Statistical methods for rates and proportions , 1973 .

[19]  Juan Merlo,et al.  Appropriate assessment of neighborhood effects on individual health: integrating random and fixed effects in multilevel logistic regression. , 2005, American journal of epidemiology.

[20]  Chuntao Wu,et al.  Do Hospitals and Surgeons With Higher Coronary Artery Bypass Graft Surgery Volumes Still Have Lower Risk-Adjusted Mortality Rates? , 2003, Circulation.

[21]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[22]  J M Robins,et al.  The role of model selection in causal inference from nonexperimental data. , 1986, American journal of epidemiology.

[23]  T. Shakespeare,et al.  Observational Studies , 2003 .

[24]  E L Hannan,et al.  New York State's Cardiac Surgery Reporting System: four years later. , 1994, The Annals of thoracic surgery.

[25]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[26]  Donald B. Rubin,et al.  Comment : Neyman ( 1923 ) and Causal Inference in Experiments and Observational Studies , 2007 .

[27]  D. Rubin,et al.  Estimating the Effects Caused by Treatments , 1984 .

[28]  Sharon-Lise T. Normand,et al.  Statistical and Clinical Aspects of Hospital Outcomes Profiling , 2007, 0710.4622.

[29]  J. Allison.  Risk adjustment for measuring health care outcomes , 1996 .

[30]  S. Greenland,et al.  Estimating causal effects. , 2002, International journal of epidemiology.

[31]  Paul R Rosenbaum,et al.  Rare Outcomes, Common Treatments: Analytic Strategies Using Propensity Scores , 2002, Annals of Internal Medicine.

[32]  Jason Goh.  Quality of care. , 2010, Clinical medicine.

[33]  Simon Day,et al.  Statistical analysis of performance indicators in UK higher education , 2005 .

[34]  Joseph Hilbe,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2009 .

[35]  Francesca Dominici,et al.  Application of a propensity score approach for risk adjustment in profiling multiple physician groups on asthma care. , 2005, Health services research.

[36]  M. Glickman,et al.  Statistical Methods for Profiling Providers of Medical Care: Issues and Applications , 1997 .

[37]  D. Rubin.  Direct and Indirect Causal Effects via Potential Outcomes * , 2004 .

[38]  Alastair Baker,et al.  Crossing the Quality Chasm: A New Health System for the 21st Century , 2001, BMJ : British Medical Journal.

[39]  R. D'Agostino.  Adjustment Methods: Propensity Score Methods for Bias Reduction in the Comparison of a Treatment to a Non‐Randomized Control Group , 2005 .

[40]  T. Osler,et al.  Use of a Matching Algorithm to Evaluate Hospital Coronary Artery Bypass Grafting Performance as an Alternative to Conventional Risk Adjustment , 2007, Medical care.

[41]  Sharon-Lise T Normand,et al.  Massachusetts cardiac surgery report card: implications of statistical methodology. , 2005, The Annals of thoracic surgery.

[42]  S L Normand,et al.  Cardiac surgery report cards: comprehensive review and statistical critique. , 2001, The Annals of thoracic surgery.

[43]  Profiling Providers of Medical Care , 2005 .

[44]  F. Grover,et al.  Cardiac surgery risk models: a position article. , 2004, The Annals of thoracic surgery.

[45]  K. Larsen,et al.  Interpreting Parameters in the Logistic Regression Model with Random Effects , 2000, Biometrics.

[46]  D. Rubin.  [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[47]  S. Greenland,et al.  Causation and causal inference in epidemiology. , 2005, American journal of public health.

[48]  P. Rosenbaum,et al.  Invited commentary: propensity scores. , 1999, American journal of epidemiology.

[49]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[50]  D. Rubin,et al.  Comment: Estimating the Effects Caused by Treatments , 1984 .

[51]  R J Little,et al.  Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches. , 2000, Annual review of public health.