Use of a Matching Algorithm to Evaluate Hospital Coronary Artery Bypass Grafting Performance as an Alternative to Conventional Risk Adjustment

Background:Although public reporting of hospital and physician performance is a cornerstone of the effort to improve health care quality, the optimal approach to risk adjustment is unknown. Objective:We sought to assess hospital quality using a matching algorithm based on a generalized distance metric and to compare this approach to the more traditional regression-based approach. Design/ Data Source:This was a retrospective study using the New York State (NYS) Coronary Artery Bypass Surgery Reporting System (CSRS), focusing on all patients undergoing isolated CABG surgery in NYS who were discharged in 1999 (18,116 patients). Patients from specific hospital were matched to a control group using the Mahalanobis distance. The hospitals’ expected mortality rate was calculated in 2 ways: (1) as the mortality rate of the control group or (2) as the mortality rate predicted by the NYS CABG model. Hospitals whose observed mortality rate was significantly different from their expected mortality rate (OE difference) were defined as quality outliers. Results:The 2 risk-adjustment methodologies disagreed on the outlier status of 4 of the 33 hospitals. Kappa analysis demonstrated substantial agreement between these 2 methods for identifying quality outliers: κ = 0.61. There was excellent agreement between the point estimates of the OE difference obtained using these 2 risk adjustment methodologies. Conclusion:Basing outcome assessment on either matching or regression modeling yielded similar findings on hospital ranking but only moderate level of agreement on hospital quality. The use of matching may enhance the transparency and acceptance of outcome report cards by hospitals and physicians.

[1]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[2]  Martin Roland,et al.  Linking physicians' pay to the quality of care--a major experiment in the United kingdom. , 2004, The New England journal of medicine.

[3]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[4]  A. Mushlin,et al.  Quality of care information makes a difference: an analysis of market share and price changes after publication of the New York State Cardiac Surgery Mortality Reports. , 1998, Medical care.

[5]  L I Iezzoni,et al.  'Black box' medical information systems. A technology needing assessment. , 1991, JAMA.

[6]  E. Hannan,et al.  Public release of cardiac surgery outcomes data in New York: what do New York state cardiologists think of it? , 1997, American heart journal.

[7]  Saharon Shelah,et al.  Black Boxes , 2008, 0812.0656.

[8]  C A Marrin,et al.  A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group. , 1996, JAMA.

[9]  A M Epstein,et al.  Use of public performance reports: a survey of patients undergoing cardiac surgery. , 1998, JAMA.

[10]  Francesca Dominici,et al.  Application of a propensity score approach for risk adjustment in profiling multiple physician groups on asthma care. , 2005, Health services research.

[11]  Russell L. Stogsdill,et al.  A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group. , 1996, JAMA.

[12]  W. Ghali,et al.  Statewide quality improvement initiatives and mortality after cardiac surgery. , 1997, JAMA.

[13]  Adult open heart surgery in New York State. An analysis of risk factors and hospital mortality rates. , 1991, JAMA.

[14]  A. Epstein.  Volume and outcome--it is time to move ahead. , 2002, The New England journal of medicine.

[15]  A M Epstein,et al.  Influence of cardiac-surgery performance reports on referral practices and access to care. A survey of cardiovascular specialists. , 1996, The New England journal of medicine.

[16]  J. Grossman,et al.  Crossing the Quality Chasm , 2005 .

[17]  Paul R. Rosenbaum,et al.  Comparison of Multivariate Matching Methods: Structures, Distances, and Algorithms , 1993 .

[18]  D. Asch,et al.  The unintended consequences of publicly reporting quality information. , 2005, JAMA.

[19]  Donald Rubin,et al.  Estimating Causal Effects from Large Data Sets Using Propensity Scores , 1997, Annals of Internal Medicine.

[20]  L I Iezzoni.  On opening "black boxes" and looking inside. , 1996, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[21]  S. Shortell,et al.  Crossing the quality chasm: implications for health services administration education. , 2004, The Journal of health administration education.

[22]  A Milstein,et al.  Selective referral to high-volume hospitals: estimating potentially avoidable deaths. , 2000, JAMA.

[23]  G. Imbens,et al.  Implementing Matching Estimators for Average Treatment Effects in Stata , 2004 .

[24]  E L Hannan,et al.  Coronary Artery Bypass Surgery: The Relationship Between Inhospital Mortality Rate and Surgical Volume After Controlling For Clinical Risk Factors , 1991, Medical care.

[25]  J. Tu,et al.  Impact of an acute myocardial infarction report card in Ontario, Canada. , 2003, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[26]  J. Iglehart,et al.  Linking compensation to quality--Medicare payments to physicians. , 2005, The New England journal of medicine.

[27]  D. Rubin.  BIAS REDUCTION USING MAHALANOBIS METRIC MATCHING , 1978 .

[28]  E L Hannan,et al.  Improving the outcomes of coronary artery bypass surgery in New York State. , 1994, JAMA.

[29]  R. D'Agostino.  Adjustment Methods: Propensity Score Methods for Bias Reduction in the Comparison of a Treatment to a Non‐Randomized Control Group , 2005 .

[30]  T. Osler,et al.  Identifying quality outliers in a large, multiple-institution database by using customized versions of the Simplified Acute Physiology Score II and the Mortality Probability Model II0* , 2002, Critical care medicine.

[31]  D. Rubin,et al.  Constructing a Control Group Using Multivariate Matched Sampling Methods That Incorporate the Propensity Score , 1985 .

[32]  D. Rubin,et al.  Reducing Bias in Observational Studies Using Subclassification on the Propensity Score , 1984 .

[33]  L I Iezzoni,et al.  The risks of risk adjustment. , 1997, JAMA.

[34]  P. Maurette,et al.  [To err is human: building a safer health system]. , 2002, Annales francaises d'anesthesie et de reanimation.

[35]  L. Kohn,et al.  To Err Is Human : Building a Safer Health System , 2007 .

[36]  A. Peitzman,et al.  Variability in trauma center outcomes for patients with moderate intracranial injury. , 2004, The Journal of trauma.

[37]  D. Rubin,et al.  Using Multivariate Matched Sampling and Regression Adjustment to Control Bias in Observational Studies , 1978 .

[38]  Janet M. Corrigan,et al.  Envisioning the national health care quality report , 2001 .

[39]  E. Blackstone.  Comparing apples and oranges. , 2002, The Journal of thoracic and cardiovascular surgery.