Do observational studies using propensity score methods agree with randomized trials? A systematic comparison of studies on acute coronary syndromes.

AIMS Randomized controlled trials (RCTs) are the gold standard for assessing the efficacy of therapeutic interventions because randomization protects from biases inherent in observational studies. Propensity score (PS) methods, proposed as a potential solution to confounding of the treatment-outcome association, are widely used in observational studies of therapeutic interventions for acute coronary syndromes (ACS). We aimed to systematically assess agreement between observational studies using PS methods and RCTs on therapeutic interventions for ACS. METHODS AND RESULTS We searched for observational studies of interventions for ACS that used PS methods to estimate treatment effects on short- or long-term mortality. Using a standardized algorithm, we matched observational studies to RCTs based on patients' characteristics, interventions, and outcomes ('topics'), and we compared estimates of treatment effect between the two designs. When multiple observational studies or RCTs were identified for the same topic, we performed a meta-analysis and used the summary relative risk for comparisons. We matched 21 observational studies investigating 17 distinct clinical topics to 63 RCTs (median = 3 RCTs per observational study) for short-term (7 topics) and long-term (10 topics) mortality. Estimates from PS analyses differed statistically significantly from randomized evidence in two instances; however, observational studies reported more extreme beneficial treatment effects compared with RCTs in 13 of 17 instances (P = 0.049). Sensitivity analyses limited to large RCTs, and using alternative meta-analysis models yielded similar results. CONCLUSION For the treatment of ACS, observational studies using PS methods produce treatment effect estimates that are of more extreme magnitude compared with those from RCTs, although the differences are rarely statistically significant.

[1]  Hani Jneid,et al.  2012 ACCF/AHA focused update of the guideline for the management of patients with unstable angina/non-ST-elevation myocardial infarction (updating the 2007 guideline and replacing the 2011 focused update): a report of the American College of Cardiology Foundation/American Heart Association Task Forc , 2012, Journal of the American College of Cardiology.

[2]  J. Ioannidis,et al.  Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials , 2011, BMJ : British Medical Journal.

[3]  Peter Jüni,et al.  An overview of the objectives of and the approaches to propensity score analyses. , 2011, European heart journal.

[4]  R. Califf,et al.  2011 ACCF/AHA Focused Update of the Guidelines for the Management of Patients With Unstable Angina/ Non–ST-Elevation Myocardial Infarction (Updating the 2007 Guideline): A Report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines , 2011, Circulation.

[5]  D. Mozaffarian,et al.  Heart disease and stroke statistics--2011 update: a report from the American Heart Association. , 2011, Circulation.

[6]  Grant D. Huang,et al.  Observational methods in comparative effectiveness research. , 2010, The American journal of medicine.

[7]  Sebastian Schneeweiss,et al.  Multivariate‐adjusted pharmacoepidemiologic analyses of confidential information pooled from multiple health care utilization databases , 2010, Pharmacoepidemiology and drug safety.

[8]  Thomas A Trikalinos,et al.  Therapeutic innovations, diminishing returns, and control rate preservation. , 2009, JAMA.

[9]  J. Avorn,et al.  High-dimensional Propensity Score Adjustment in Studies of Treatment Effects Using Health Care Claims Data , 2009, Epidemiology.

[10]  Peter C Austin,et al.  Report Card on Propensity-Score Matching in the Cardiology Literature From 2004 to 2006: A Systematic Review , 2008, Circulation. Cardiovascular quality and outcomes.

[11]  Peter C Austin,et al.  A critical appraisal of propensity‐score matching in the medical literature between 1996 and 2003 , 2008, Statistics in medicine.

[12]  Peter C Austin,et al.  Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study , 2007, Statistics in medicine.

[13]  Elliott S Fisher,et al.  Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. , 2007, JAMA.

[14]  D. Rubin.  The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials , 2007, Statistics in medicine.

[15]  Andrew J. Krainik,et al.  The Washington Manual Of Medical Therapeutics , 2007 .

[16]  Til Stürmer,et al.  A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. , 2006, Journal of clinical epidemiology.

[17]  Peter C Austin,et al.  Propensity score methods gave similar results to traditional regression modeling in observational studies: a systematic review. , 2005, Journal of clinical epidemiology.

[18]  P. Armstrong,et al.  Association of revascularisation with low mortality in non-ST elevation acute coronary syndrome, a report from GUSTO IV-ACS. , 2004, European heart journal.

[19]  M Soledad Cepeda,et al.  Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders. , 2003, American journal of epidemiology.

[20]  Lawrence Joseph,et al.  Evidence for use of coronary stents: a hierarchical bayesian meta-analysis ☆ , 2003 .

[21]  Lawrence Joseph,et al.  Evidence for Use of Coronary Stents , 2003 .

[22]  Carl J Pepine,et al.  ACC/AHA guideline update for the management of patients with unstable angina and non-ST-segment elevation myocardial infarction--2002: summary article: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on the Management of Patients , 2002, Circulation.

[23]  G. Rongen.  The Washington Manual of Medical Therapeutics , 2002 .

[24]  J. Ioannidis,et al.  Comparison of evidence of treatment effects in randomized and nonrandomized studies. , 2001, JAMA.

[25]  M Michele,et al.  Any casualties in the clash of randomised and observational evidence , 2001 .

[26]  J. Concato,et al.  Randomized, controlled trials, observational studies, and the hierarchy of research designs. , 2000, The New England journal of medicine.

[27]  A. Hartz,et al.  A comparison of observational studies and randomized, controlled trials , 2000, American journal of ophthalmology.

[28]  S J Pocock,et al.  Randomized trials or observational tribulations? , 2000, The New England journal of medicine.

[29]  D. Rubin,et al.  Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates , 2000 .

[30]  H. Williams,et al.  Systematic review of treatments for atopic eczema. , 2000, Health technology assessment.

[31]  R J Little,et al.  Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches. , 2000, Annual review of public health.

[32]  T A Sheldon,et al.  A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies. , 2000, Health technology assessment.

[33]  S. Casswell,et al.  What is meant by intention to treat analysis? Survey of published randomised controlled trials , 1999 .

[34]  T. Shakespeare,et al.  Observational Studies , 2003 .

[35]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[36]  R Peto,et al.  Why do we need some large, simple randomized trials? , 1984, Statistics in medicine.

[37]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[38]  D. Byar.  Why data bases should not replace randomized clinical trials. , 1980, Biometrics.