Network meta-analysis incorporating randomized controlled trials and non-randomized comparative cohort studies for assessing the safety and effectiveness of medical treatments: challenges and opportunities

Network meta-analysis is increasingly used to allow comparison of multiple treatment alternatives simultaneously, some of which may not have been compared directly in primary research studies. The majority of network meta-analyses published to date have incorporated data from randomized controlled trials (RCTs) only; however, inclusion of non-randomized studies may sometimes be considered. Non-randomized studies can complement RCTs or address some of their limitations, such as short follow-up time, small sample size, highly selected population, high cost, and ethical restrictions. In this paper, we discuss the challenges and opportunities of incorporating both RCTs and non-randomized comparative cohort studies into network meta-analysis for assessing the safety and effectiveness of medical treatments. Non-randomized studies with inadequate control of biases such as confounding may threaten the validity of the entire network meta-analysis. Therefore, identification and inclusion of non-randomized studies must balance their strengths with their limitations. Inclusion of both RCTs and non-randomized studies in network meta-analysis will likely increase in the future due to the growing need to assess multiple treatments simultaneously, the availability of higher quality non-randomized data and more valid methods, and the increased use of progressive licensing and product listing agreements requiring collection of data over the life cycle of medical products. Inappropriate inclusion of non-randomized studies could perpetuate the biases that are unknown, unmeasured, or uncontrolled. However, thoughtful integration of randomized and non-randomized studies may offer opportunities to provide more timely, comprehensive, and generalizable evidence about the comparative safety and effectiveness of medical treatments.

[1]  P. Ravani,et al.  Proton pump inhibitors and the risk of hospitalisation for community-acquired pneumonia: replicated cohort studies with meta-analysis , 2013, Gut.

[2]  T. Stijnen,et al.  Different combined oral contraceptives and the risk of venous thrombosis: systematic review and network meta-analysis , 2013, BMJ : British Medical Journal.

[3]  S Shapiro,et al.  Confounding by indication? , 1997, Epidemiology.

[4]  Bruce M Psaty,et al.  Benefits and risks of drug treatments: how to combine the best evidence on benefits with the best data about adverse effects. , 2008, JAMA.

[5]  H. Eichler,et al.  Adaptive Licensing: Taking the Next Step in the Evolution of Drug Approval , 2012, Clinical pharmacology and therapeutics.

[6]  M. Epstein,et al.  Guidelines for good pharmacoepidemiology practices (GPP) , 2008 .

[7]  Cathal Walsh,et al.  Incorporating data from various trial designs into a mixed treatment comparison model , 2013, Statistics in medicine.

[8]  Mohammad Hassan Murad,et al.  A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis , 2014, BMJ : British Medical Journal.

[9]  J. Concato,et al.  Randomized, controlled trials, observational studies, and the hierarchy of research designs. , 2000, The New England journal of medicine.

[10]  Kristian Thorlund,et al.  The PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions: Checklist and Explanations , 2015, Annals of Internal Medicine.

[11]  J. Ioannidis,et al.  Comparison of evidence of treatment effects in randomized and nonrandomized studies. , 2001, JAMA.

[12]  S. Pocock,et al.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. , 2007, Preventive medicine.

[13]  K. Rothman,et al.  Beyond randomized controlled trials: A critical comparison of trials with nonrandomized studies , 2006, Hepatology.

[14]  M. Hernán,et al.  Beyond the intention-to-treat in comparative effectiveness research , 2012, Clinical trials.

[15]  R. Platt,et al.  Distributed Health Data Networks: A Practical and Preferred Approach to Multi-Institutional Evaluations of Comparative Effectiveness, Safety, and Quality of Care , 2010, Medical care.

[16]  F. Song,et al.  Evaluating non-randomised intervention studies. , 2003, Health technology assessment.

[17]  Deborah M Caldwell,et al.  Simultaneous comparison of multiple treatments: combining direct and indirect evidence , 2005, BMJ : British Medical Journal.

[18]  K. Schulz,et al.  Bias and causal associations in observational research , 2002, The Lancet.

[19]  Reshma Jagsi,et al.  Coordinating Cancer Care: Patient and Practice Management Processes Among Surgeons Who Treat Breast Cancer , 2010, Medical care.

[20]  N. Pratt,et al.  The Asian Pharmacoepidemiology Network (AsPEN): promoting multi‐national collaboration for pharmacoepidemiologic research in Asia , 2013, Pharmacoepidemiology and drug safety.

[21]  Tommi Tervonen,et al.  ADDIS: A decision support system for evidence-based medicine , 2013, Decis. Support Syst..

[22]  S. Pocock,et al.  Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration , 2007, PLoS medicine.

[23]  James M. Robins,et al.  Observational Studies Analyzed Like Randomized Experiments: An Application to Postmenopausal Hormone Therapy and Coronary Heart Disease , 2008, Epidemiology.

[24]  Ron Goeree,et al.  Bayesian Hierarchical Models Combining Different Study Types and Adjusting for Covariate Imbalances: A Simulation Study to Assess Model Performance , 2011, PloS one.

[25]  S. Morgan,et al.  Inter-jurisdictional cooperation on pharmaceutical product listing agreements: views from Canadian provinces , 2013, BMC Health Services Research.

[26]  Sengwee Toh,et al.  A protocol for active surveillance of acute myocardial infarction in association with the use of a new antidiabetic pharmaceutical agent , 2012, Pharmacoepidemiology and drug safety.

[27]  A. Levy,et al.  CNODES: the Canadian Network for Observational Drug Effect Studies , 2012, Open medicine : a peer-reviewed, independent, open-access journal.

[28]  John P.A. Ioannidis,et al.  Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses , 2009, Canadian Medical Association Journal.

[29]  M. Schuemie,et al.  Combining electronic healthcare databases in Europe to allow for large‐scale drug safety monitoring: the EU‐ADR Project , 2011, Pharmacoepidemiology and drug safety.

[30]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[31]  Sengwee Toh,et al.  An Analytic Framework for Aligning Observational and Randomized Trial Data: Application to Postmenopausal Hormone Therapy and Coronary Heart Disease , 2013, Statistics in biosciences.

[32]  Christopher H. Schmid,et al.  Characteristics of Networks of Interventions: A Description of a Database of 186 Published Networks , 2014, PloS one.

[33]  Christian Ohmann,et al.  Combining randomized and non‐randomized evidence in clinical research: a review of methods and applications , 2015, Research synthesis methods.

[34]  R. Platt,et al.  Developing the Sentinel System--a national resource for evidence development. , 2011, The New England journal of medicine.

[35]  Huseyin Naci,et al.  Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers , 2013, BMC Medicine.

[36]  Adrian F Hernandez,et al.  Comparative risk for angioedema associated with the use of drugs that target the renin-angiotensin-aldosterone system. , 2012, Archives of internal medicine.

[37]  Douglas G Altman,et al.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. , 2014, International journal of surgery.

[38]  Kristian B Filion,et al.  Higher potency statins and the risk of new diabetes: multicentre, observational study of administrative databases , 2014, BMJ : British Medical Journal.

[39]  R. Goeree,et al.  The importance of adjusting for potential confounders in Bayesian hierarchical models synthesising evidence from randomised and non-randomised studies: an application comparing treatments for abdominal aortic aneurysms , 2010, BMC medical research methodology.

[40]  Luke Vale,et al.  Relative effectiveness of robot‐assisted and standard laparoscopic prostatectomy as alternatives to open radical prostatectomy for treatment of localised prostate cancer: a systematic review and mixed treatment comparison meta‐analysis , 2013, BJU international.

[41]  Alex J. Sutton,et al.  Evidence Synthesis for Decision Making 2 , 2013, Medical decision making : an international journal of the Society for Medical Decision Making.

[42]  A. Garg,et al.  Use of high potency statins and rates of admission for acute kidney injury: multicenter, retrospective observational analysis of administrative databases , 2013, BMJ.

[43]  Alex J. Sutton,et al.  Evidence Synthesis for Decision Making Inconsistency in Networks of Evidence Based on Randomized Controlled Trials. , 2013 .

[44]  N. Freemantle,et al.  Making inferences on treatment effects from real world data: propensity scores, confounding by indication, and other perils for the unwary in observational research , 2013, BMJ.

[45]  D. Holmes,et al.  Clinical ResearchInterventional CardiologyCulprit Vessel Only Versus Multivessel and Staged Percutaneous Coronary Intervention for Multivessel Disease in Patients Presenting With ST-Segment Elevation Myocardial Infarction: A Pairwise and Network Meta-Analysis , 2011 .

[46]  J. Robins,et al.  Results of multivariable logistic regression, propensity matching, propensity adjustment, and propensity-based weighting under conditions of nonuniform effect. , 2006, American journal of epidemiology.

[47]  K R Abrams,et al.  Bayesian methods in meta-analysis and evidence synthesis. , 2001, Statistical methods in medical research.

[48]  C. Mazer,et al.  Risks of harms using antifibrinolytics in cardiac surgery: systematic review and network meta-analysis of randomised and observational studies , 2012, BMJ : British Medical Journal.