Sample size and power considerations in network meta-analysis

BackgroundNetwork meta-analysis is becoming increasingly popular for establishing comparative effectiveness among multiple interventions for the same disease. Network meta-analysis inherits all methodological challenges of standard pairwise meta-analysis, but with increased complexity due to the multitude of intervention comparisons. One issue that is now widely recognized in pairwise meta-analysis is the issue of sample size and statistical power. This issue, however, has so far only received little attention in network meta-analysis. To date, no approaches have been proposed for evaluating the adequacy of the sample size, and thus power, in a treatment network.FindingsIn this article, we develop easy-to-use flexible methods for estimating the ‘effective sample size’ in indirect comparison meta-analysis and network meta-analysis. The effective sample size for a particular treatment comparison can be interpreted as the number of patients in a pairwise meta-analysis that would provide the same degree and strength of evidence as that which is provided in the indirect comparison or network meta-analysis. We further develop methods for retrospectively estimating the statistical power for each comparison in a network meta-analysis. We illustrate the performance of the proposed methods for estimating effective sample size and statistical power using data from a network meta-analysis on interventions for smoking cessation including over 100 trials.ConclusionThe proposed methods are easy to use and will be of high value to regulatory agencies and decision makers who must assess the strength of the evidence supporting comparative effectiveness estimates.

[1]  A Whitehead,et al.  Borrowing strength from external trials in a meta-analysis. , 1996, Statistics in medicine.

[2]  K. Thorlund,et al.  Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses. , 2008, Journal of clinical epidemiology.

[3]  Julian P T Higgins,et al.  Recent developments in meta‐analysis , 2008, Statistics in medicine.

[4]  G. Guyatt,et al.  GRADE guidelines 6. Rating the quality of evidence--imprecision. , 2011, Journal of clinical epidemiology.

[5]  J. Higgins.  Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration , 2011 .

[6]  Kristian Thorlund,et al.  Estimating the Power of Indirect Comparisons: A Simulation Study , 2011, PloS one.

[7]  A Whitehead,et al.  A general parametric approach to the meta-analysis of randomized clinical trials. , 1991, Statistics in medicine.

[8]  Keith Abrams,et al.  Use of Indirect and Mixed Treatment Comparisons for Technology Assessment , 2012, PharmacoEconomics.

[9]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration , 2013 .

[10]  Kristian Thorlund,et al.  Clinical Epidemiology Dovepress , 2022 .

[11]  J. Ioannidis,et al.  Evolution of treatment effects over time: empirical insight from recursive cumulative metaanalyses. , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[12]  S D Walter,et al.  The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. , 1997, Journal of clinical epidemiology.

[13]  Kristian Thorlund,et al.  Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses? , 2009, International journal of epidemiology.

[14]  K. Thorlund,et al.  Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. , 2008, Journal of clinical epidemiology.

[15]  Douglas G Altman,et al.  Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews , 2009, BMJ : British Medical Journal.

[16]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[17]  John P A Ioannidis,et al.  Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. , 2011, Journal of clinical epidemiology.

[18]  T. Lumley.  Network meta‐analysis for indirect treatment comparisons , 2002, Statistics in medicine.

[19]  D G Altman,et al.  Indirect comparisons of competing interventions. , 2005, Health technology assessment.

[20]  Kristian Thorlund,et al.  Evolution of Heterogeneity (I2) Estimates and Their 95% Confidence Intervals in Large Meta-Analyses , 2012, PloS one.

[21]  S. Yusuf,et al.  Overcoming the limitations of current meta-analysis of randomised controlled trials , 1998, The Lancet.

[22]  Kristian Thorlund,et al.  Apparently conclusive meta-analyses may be inconclusive--Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses. , 2009, International journal of epidemiology.

[23]  Michele Tarsilla.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[24]  K. Thorlund,et al.  Comments on ‘Sequential meta-analysis: an efficient decision-making tool’ by I van der Tweel and C Bollen , 2010, Clinical trials.

[25]  Douglas G Altman,et al.  Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses , 2003, BMJ : British Medical Journal.

[26]  G. Lu,et al.  Combination of direct and indirect evidence in mixed treatment comparisons , 2004, Statistics in medicine.

[27]  M. Puhan,et al.  Comparisons of high-dose and combination nicotine replacement therapy, varenicline, and bupropion for smoking cessation: A systematic review and multiple treatment meta-analysis , 2012, Annals of medicine.

[28]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[29]  Kristian Thorlund,et al.  Multiple treatment comparison meta-analyses: a step forward into complexity , 2011, Clinical epidemiology.

[30]  John P.A. Ioannidis,et al.  Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses , 2009, Canadian Medical Association Journal.

[31]  Kristian Thorlund,et al.  The Number of Patients and Events Required to Limit the Risk of Overestimation of Intervention Effects in Meta-Analysis—A Simulation Study , 2011, PloS one.

[32]  Alex J Sutton,et al.  Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study , 2011, BMJ : British Medical Journal.

[33]  Jeroen P Jansen,et al.  Bayesian meta-analysis of multiple treatment comparisons: an introduction to mixed treatment comparisons. , 2008, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[34]  Georgia Salanti,et al.  Evaluation of networks of randomized trials , 2008, Statistical methods in medical research.

[35]  Anne Whitehead,et al.  Sequential methods for random-effects meta-analysis , 2010, Statistics in medicine.

[36]  I. van der Tweel,et al.  Sequential meta-analysis: an efficient decision-making tool , 2010, Clinical trials.

[37]  Kristian Thorlund,et al.  Estimating required information size by quantifying diversity in random-effects model meta-analyses , 2009, BMC medical research methodology.

[38]  Kristian Thorlund,et al.  How to use an article reporting a multiple treatment comparison meta-analysis. , 2012, JAMA.

[39]  S. Yusuf,et al.  Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. , 1997, Controlled clinical trials.