Selecting Observational Studies for Comparing Medical Interventions

Comparative Effectiveness Reviews are systematic reviews of existing research on the effectiveness, comparative effectiveness, and harms of different health care interventions. They provide syntheses of relevant evidence to inform real-world health care decisions for patients, providers, and policymakers. Strong methodologic approaches to systematic review improve the transparency, consistency, and scientific rigor of these reports. Through a collaborative effort of the Effective Health Care (EHC) Program, the Agency for Healthcare Research and Quality (AHRQ), the EHC Program Scientific Resource Center, and the AHRQ Evidence-based Practice Centers have developed a Methods Guide for Comparative Effectiveness Reviews. This Guide presents issues key to the development of Comparative Effectiveness Reviews and describes recommended approaches for addressing difficult, frequently encountered methodological issues. The Methods Guide for Comparative Effectiveness Reviews is a living document, and will be updated as further empiric evidence develops and our understanding of better methods improves. Comments and suggestions on the Methods Guide for Comparative Effectiveness Reviews and the Effective Health Care Program can be made at www.effectivehealthcare.ahrq.gov. None of the authors has a financial interest in any of the products discussed in this document. The views expressed in this paper are those of the authors and do not represent the official • Systematic reviewers disagree about the ability of observational studies to answer questions about the benefits or intended effects of pharmacotherapeutic, device, or procedural interventions. • This paper provides a framework for decisionmaking on the inclusion of observational studies to assess benefits and intended effects in comparative effectiveness reviews • Comparative effectiveness reviewers should routinely assess the appropriateness of inclusion of observational studies for questions of benefit, and the rationale for inclusion or exclusion of such studies should be explicitly stated in reviews. • In considering whether to use observational studies in CERs for addressing beneficial effects, reviewers should answer two questions: o Are there gaps in the evidence from randomized controlled trials? o Will observational studies provide valid and useful information?

[1]  G. Guyatt,et al.  Grading quality of evidence and strength of recommendations , 2004, BMJ : British Medical Journal.

[2]  W. Ray,et al.  Evaluating medication effects outside of clinical trials: new-user designs. , 2003, American journal of epidemiology.

[3]  P. Rothwell,et al.  External validity of randomised controlled trials: “To whom do the results of this trial apply?” , 2005, The Lancet.

[4]  S. Pocock,et al.  Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies , 2007, BMJ : British Medical Journal.

[5]  J. Avorn,et al.  A review of uses of health care utilization databases for epidemiologic research on therapeutics. , 2005, Journal of clinical epidemiology.

[6]  M. Helfand.  Drug Class Review on the Triptans , 2009 .

[7]  Jan P Vandenbroucke,et al.  When are observational studies as credible as randomised trials? , 2004, The Lancet.

[8]  David Moher,et al.  AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program. , 2010, Journal of clinical epidemiology.

[9]  B. Chan,et al.  Drug Class Review: Atypical Antipsychotic Drugs: Final Report Update 2 , 2008 .

[10]  M. Etminan,et al.  Challenges and Opportunities for Pharmacoepidemiology in Drug‐Therapy Decision Making , 2006, Journal of clinical pharmacology.

[11]  K. Lohr,et al.  A simple and valid tool distinguished efficacy from effectiveness studies. , 2006, Journal of clinical epidemiology.

[12]  D. Altman,et al.  Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research , 2004, Canadian Medical Association Journal.

[13]  P. Williamson,et al.  Bias in meta‐analysis due to outcome variable selection within studies , 2000 .

[14]  Alessandro Liberati,et al.  Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study , 2005, BMJ : British Medical Journal.

[15]  N. Black.  Why we need observational studies to evaluate the effectiveness of health care , 1996, BMJ.

[16]  K. Mengersen,et al.  Selective Reporting of Adjusted Estimates in Observational Epidemiology Studies: Reasons and Implications for Meta-analyses , 2008, Evaluation & the health professions.

[17]  Howard Balshem,et al.  AHRQ series paper 2: principles for developing guidance: AHRQ and the effective health-care program. , 2010, Journal of clinical epidemiology.

[18]  AndrewJ. S. Coats.  MRC/BHF Heart Protection Study of cholesterol lowering with simvastatin in 20 536 high-risk individuals: a randomised placebocontrolled trial , 2002, The Lancet.

[19]  Michele Tarsilla.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[20]  N McKoy,et al.  Systems to rate the strength of scientific evidence. , 2002, Evidence report/technology assessment.

[21]  F. Song,et al.  Evaluating non-randomised intervention studies. , 2003, Health technology assessment.

[22]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[23]  Douglas K Owens,et al.  AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions--agency for healthcare research and quality and the effective health-care program. , 2010, Journal of clinical epidemiology.

[24]  A. Go,et al.  Comparative effectiveness of different beta-adrenergic antagonists on mortality among adults with heart failure in clinical practice. , 2008, Archives of Internal Medicine.

[25]  Paul Glasziou,et al.  When are randomised trials unnecessary? Picking signal from noise , 2007, BMJ : British Medical Journal.

[26]  P. Raina,et al.  The uses of heparin to treat burn injury. , 2006, Evidence report/technology assessment.

[27]  G. Guyatt,et al.  GRADE: an emerging consensus on rating quality of evidence and strength of recommendations , 2008, BMJ : British Medical Journal.

[28]  G. Guyatt,et al.  Association between unreported outcomes and effect size estimates in Cochrane meta-analyses. , 2007, JAMA.

[29]  Jonathan J Deeks,et al.  Chapter 13: Including non-randomized studies , 2008 .

[30]  A. Laupacis,et al.  Gaps in the evaluation and monitoring of new pharmaceuticals: proposal for a different approach. , 2003, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[31]  D. Atkins,et al.  Assessing applicability when comparing medical interventions: AHRQ and the Effective Health Care Program. , 2011, Journal of clinical epidemiology.

[32]  David Atkins,et al.  Challenges in Using Nonrandomized Studies in Systematic Reviews of Treatment Interventions , 2005, Annals of Internal Medicine.

[33]  Moira C McKinnon,et al.  Comparative Effectiveness of Percutaneous Coronary Interventions and Coronary Artery Bypass Grafting for Coronary Artery Disease , 2007 .