Simulation-based power calculations for planning a two-stage individual participant data meta-analysis

BackgroundResearchers and funders should consider the statistical power of planned Individual Participant Data (IPD) meta-analysis projects, as they are often time-consuming and costly. We propose simulation-based power calculations utilising a two-stage framework, and illustrate the approach for a planned IPD meta-analysis of randomised trials with continuous outcomes where the aim is to identify treatment-covariate interactions.MethodsThe simulation approach has four steps: (i) specify an underlying (data generating) statistical model for trials in the IPD meta-analysis; (ii) use readily available information (e.g. from publications) and prior knowledge (e.g. number of studies promising IPD) to specify model parameter values (e.g. control group mean, intervention effect, treatment-covariate interaction); (iii) simulate an IPD meta-analysis dataset of a particular size from the model, and apply a two-stage IPD meta-analysis to obtain the summary estimate of interest (e.g. interaction effect) and its associated p-value; (iv) repeat the previous step (e.g. thousands of times), then estimate the power to detect a genuine effect by the proportion of summary estimates with a significant p-value.ResultsIn a planned IPD meta-analysis of lifestyle interventions to reduce weight gain in pregnancy, 14 trials (1183 patients) promised their IPD to examine a treatment-BMI interaction (i.e. whether baseline BMI modifies intervention effect on weight gain). Using our simulation-based approach, a two-stage IPD meta-analysis has < 60% power to detect a reduction of 1 kg weight gain for a 10-unit increase in BMI. Additional IPD from ten other published trials (containing 1761 patients) would improve power to over 80%, but only if a fixed-effect meta-analysis was appropriate. Pre-specified adjustment for prognostic factors would increase power further. Incorrect dichotomisation of BMI would reduce power by over 20%, similar to immediately throwing away IPD from ten trials.ConclusionsSimulation-based power calculations could inform the planning and funding of IPD projects, and should be used routinely.

[1]  Evangelos Kontopantelis,et al.  Simulation-Based Power Calculations for Mixed Effects Modeling: ipdpower in Stata , 2016 .

[2]  Michael J Crowther,et al.  Simulating biologically plausible complex survival data , 2013, Statistics in medicine.

[3]  Raghu Kacker,et al.  Random-effects model for meta-analysis of clinical trials: an update. , 2007, Contemporary clinical trials.

[4]  Stephanie A Kovalchik,et al.  Using aggregate data to estimate the standard error of a treatment–covariate interaction in an individual patient data meta‐analysis , 2012, Biometrical journal. Biometrische Zeitschrift.

[5]  Kurex Sidik,et al.  A simple confidence interval for meta‐analysis , 2002, Statistics in medicine.

[6]  Mark Simmonds,et al.  A decade of individual participant data meta-analyses: A review of current practice. , 2015, Contemporary clinical trials.

[7]  R. Riley,et al.  Meta-analysis of individual participant data: rationale, conduct, and reporting , 2010, BMJ : British Medical Journal.

[8]  Sara T Brookes,et al.  Subgroup analyses in randomized trials: risks of subgroup-specific analyses; power and sample size for the interaction test. , 2004, Journal of clinical epidemiology.

[9]  D. Altman,et al.  Analysing controlled trials with baseline and follow up measurements , 2001, BMJ : British Medical Journal.

[10]  Joachim Hartung,et al.  An Alternative Method for Meta‐Analysis , 1999 .

[11]  J. Hartung,et al.  A refined method for the meta‐analysis of controlled clinical trials with binary outcome , 2001, Statistics in medicine.

[12]  Richard D Riley,et al.  Prognosis research strategy (PROGRESS) 4: Stratified medicine research , 2013, BMJ : British Medical Journal.

[13]  Richard D. Riley,et al.  Random effects meta‐analysis: Coverage performance of 95% confidence and prediction intervals following REML estimation , 2016, Statistics in medicine.

[14]  Alan E Hubbard,et al.  Simulation methods to estimate design power: an overview for applied research , 2011, BMC medical research methodology.

[15]  A Whitehead,et al.  A general parametric approach to the meta-analysis of randomized clinical trials. , 1991, Statistics in medicine.

[16]  E. S. Pearson,et al.  THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL , 1934 .

[17]  S. Kovalchik.  Aggregate-data estimation of an individual patient data linear random effects meta-analysis with a patient covariate-treatment interaction term. , 2013, Biostatistics.

[18]  A. Hrõbjartsson.  Why did it take 19 months to retrieve clinical trial data from a non-profit organisation? , 2013, BMJ.

[19]  Guido Knapp,et al.  Improved tests for a random effects meta‐regression with a single covariate , 2003, Statistics in medicine.

[20]  Ralf Bender,et al.  Methods to estimate the between‐study variance and its uncertainty in meta‐analysis† , 2015, Research synthesis methods.

[21]  Douglas G Altman,et al.  Dichotomizing continuous predictors in multiple regression: a bad idea , 2006, Statistics in medicine.

[22]  Mark C Simmonds,et al.  Meta-analysis of individual patient data from randomized trials: a review of methods used in practice , 2005, Clinical trials.

[23]  Sabine Landau,et al.  Sample size and power calculations for medical studies by simulation when closed form expressions are not available , 2013, Statistical methods in medical research.

[24]  A. Harris,et al.  Systematic Review of Multiple Studies of Prognosis: The Feasibility of Obtaining Individual Patient Data , 2007 .

[25]  Evangelos Kontopantelis,et al.  Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation study , 2012, Statistical methods in medical research.

[26]  Patrick Royston,et al.  The cost of dichotomising continuous variables , 2006, BMJ : British Medical Journal.

[27]  B W Mol,et al.  Effects of interventions in pregnancy on maternal weight and obstetric outcomes: meta-analysis of randomised evidence , 2012, BMJ : British Medical Journal.

[28]  Richard D Riley,et al.  Meta‐analysis of randomised trials with a continuous outcome according to baseline imbalance and availability of individual participant data , 2013, Statistics in medicine.

[29]  Richard D Riley,et al.  Individual participant data meta-analysis of prognostic factor studies: state of the art? , 2012, BMC Medical Research Methodology.

[30]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, International Coaching Psychology Review.

[31]  Richard D Riley,et al.  Meta‐analysis using individual participant data: one‐stage and two‐stage approaches, and why they may differ , 2016, Statistics in medicine.

[32]  M C Simmonds,et al.  Covariate heterogeneity in meta‐analysis: Criteria for deciding between meta‐regression and individual patient data , 2007, Statistics in medicine.

[33]  William J. Browne,et al.  A guide to sample size calculations for random effect models via simulation and the MLPowSim Software Package , 2009 .

[34]  Kurex Sidik,et al.  On Constructing Confidence Intervals for a Standardized Mean Difference in Meta-analysis , 2003 .

[35]  A. H. Feiveson,et al.  Power by Simulation , 2002 .

[36]  J. Hartung,et al.  On tests of the overall treatment effect in meta‐analysis with normally distributed responses , 2001, Statistics in medicine.

[37]  George F Borm,et al.  The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method , 2014, BMC Medical Research Methodology.

[38]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[39]  David Fisher,et al.  Two-stage Individual Participant Data Meta-analysis and Generalized Forest Plots , 2015 .

[40]  Tim P Morris,et al.  A comparison of methods to adjust for continuous covariates in the analysis of randomised trials , 2016, BMC Medical Research Methodology.

[41]  J F Tierney,et al.  A critical review of methods for the assessment of patient-level interactions in individual participant data meta-analysis of randomized trials, and guidance for practitioners. , 2011, Journal of clinical epidemiology.

[42]  W. Tam,et al.  Distribution and Epidemiological Characteristics of Published Individual Patient Data Meta-Analyses , 2014, PloS one.

[43]  A. Sutton,et al.  Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey , 2012, BMJ : British Medical Journal.

[44]  Richard D Riley,et al.  One‐stage individual participant data meta‐analysis models: estimation of treatment‐covariate interactions must avoid ecological bias by separating out within‐trial and across‐trial information , 2016, Statistics in medicine.

[45]  Harlan M Krumholz,et al.  Why data sharing should be the expected norm , 2015, BMJ : British Medical Journal.