Methods to estimate the between‐study variance and its uncertainty in meta‐analysis†

Meta‐analyses are typically used to estimate the overall/mean of an outcome of interest. However, inference about between‐study variability, which is typically modelled using a between‐study variance parameter, is usually an additional aim. The DerSimonian and Laird method, currently widely used by default to estimate the between‐study variance, has been long challenged. Our aim is to identify known methods for estimation of the between‐study variance and its corresponding uncertainty, and to summarise the simulation and empirical evidence that compares them. We identified 16 estimators for the between‐study variance, seven methods to calculate confidence intervals, and several comparative studies. Simulation studies suggest that for both dichotomous and continuous data the estimator proposed by Paule and Mandel and for continuous data the restricted maximum likelihood estimator are better alternatives to estimate the between‐study variance. Based on the scenarios and results presented in the published studies, we recommend the Q‐profile method and the alternative approach based on a ‘generalised Cochran between‐study variance statistic’ to compute corresponding confidence intervals around the resulting estimates. Our recommendations are based on a qualitative evaluation of the existing literature and expert consensus. Evidence‐based recommendations require an extensive simulation study where all methods would be compared under the same scenarios. © 2015 The Authors. Research Synthesis Methods published by John Wiley & Sons Ltd.

[1]  Jonathan J Deeks,et al.  Issues in the selection of a summary statistic for meta‐analysis of clinical trials with binary outcomes , 2002, Statistics in medicine.

[2]  Kurex Sidik,et al.  A Note on Variance Estimation in Random Effects Meta-Regression , 2005, Journal of biopharmaceutical statistics.

[3]  Theo Stijnen,et al.  Random effects meta‐analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data , 2010, Statistics in medicine.

[4]  A Ziegler,et al.  A Simplification and Implementation of Random-effects Meta-analyses Based on the Exact Distribution of Cochran’s Q , 2013, Methods of Information in Medicine.

[5]  Wolfgang Viechtbauer,et al.  Confidence intervals for the amount of heterogeneity in meta‐analysis , 2007, Statistics in medicine.

[6]  Raghu Kacker,et al.  Random-effects model for meta-analysis of clinical trials: an update. , 2007, Contemporary clinical trials.

[7]  Jessica Gurevitch,et al.  MetaWin: Statistical Software for Meta-analysis with Resampling Tests , 1997 .

[8]  Kurex Sidik,et al.  A comparison of heterogeneity variance estimators in combining results of studies , 2007, Statistics in medicine.

[9]  Joachim Hartung,et al.  An Alternative Method for Meta‐Analysis , 1999 .

[10]  B J Biggerstaff,et al.  Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. , 1997, Statistics in medicine.

[11]  Jonathan J Deeks,et al.  Much ado about nothing: a comparison of the performance of meta‐analytical methods with rare events , 2007, Statistics in medicine.

[12]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[13]  Guido Knapp,et al.  On confidence intervals for the among-group variance in the one-way random effects model with unequal error variances , 2005 .

[14]  Sarah E. Brockwell,et al.  A comparison of statistical methods for meta‐analysis , 2001, Statistics in medicine.

[15]  A Donner,et al.  Meta‐analyses in systematic reviews of randomized controlled trials in perinatal medicine: comparison of fixed and random effects models , 2001, Statistics in medicine.

[16]  W. G. Cochran.  The combination of estimates from different experiments. , 1954 .

[17]  Lynn Friedman,et al.  Estimators of Random Effects Variance Components in Meta-Analysis , 2000 .

[18]  Sophia Rabe-Hesketh,et al.  Maximum Likelihood Estimation of Generalized Linear Models with Covariate Measurement Error , 2003 .

[19]  Julio Sánchez-Meca,et al.  Confidence intervals for the overall effect size in random-effects meta-analysis. , 2008, Psychological methods.

[20]  Evangelos Kontopantelis,et al.  Metaan: Random-effects Meta-analysis , 2010 .

[21]  Simon G Thompson,et al.  Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews , 2012, International journal of epidemiology.

[22]  Byron C. Wallace,et al.  Closing the Gap between Methodologists and End-Users: R as a Computational Back-End , 2012 .

[23]  S. Sharp,et al.  Explaining heterogeneity in meta-analysis: a comparison of methods. , 1997, Statistics in medicine.

[24]  Fritz Drasgow,et al.  Bootstrap estimates of standard errors in validity generalization , 1992 .

[25]  Joachim Hartung,et al.  Reducing the Number of Unjustified Significant Results in Meta-analysis , 2003 .

[26]  Harvey Goldstein,et al.  Improved Approximations for Multilevel Models with Binary Responses , 1996 .

[27]  Jack Bowden,et al.  Quantifying, displaying and accounting for heterogeneity in the meta-analysis of RCTs using standard and generalised Q statistics , 2011, BMC medical research methodology.

[28]  Alan H. Welsh,et al.  Likelihood inference for small variance components , 2000 .

[29]  Pierre Lafaye de Micheaux,et al.  Computing the distribution of quadratic forms: Further comparisons between the Liu-Tang-Zhang approximation and exact methods , 2010, Comput. Stat. Data Anal..

[30]  Larry V. Hedges,et al.  A random effects model for effect sizes , 1983 .

[31]  Kit C B Roes,et al.  Estimation of between-trial variance in sequential meta-analyses: a simulation study. , 2014, Contemporary clinical trials.

[32]  Andrew L. Rukhin,et al.  Restricted maximum likelihood estimation of a common mean and the Mandel–Paule algorithm , 2000 .

[33]  Evangelos Kontopantelis,et al.  MetaEasy: A Meta-Analysis Add-In for Microsoft Excel , 2009 .

[34]  Mike W.-L. Cheung,et al.  metaSEM: an R package for meta-analysis using structural equation modeling , 2015, Front. Psychol..

[35]  Eleanor M Pullenayegum,et al.  An informed reference prior for between‐study heterogeneity in meta‐analyses of binary outcomes , 2011, Statistics in medicine.

[36]  Evangelos Kontopantelis,et al.  Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A comparison between DerSimonian–Laird and restricted maximum likelihood , 2012, Statistical methods in medical research.

[37]  Cynthia D Mulrow,et al.  Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change , 2014, Annals of Internal Medicine.

[38]  J. Hartung,et al.  Positive estimation of the between-study variance in meta-analysis : theory and methods , 2002 .

[39]  O Kuss,et al.  Statistical methods for meta‐analyses including information from studies without any events—add nothing to nothing and succeed nevertheless , 2015, Statistics in medicine.

[40]  Kristian Thorlund,et al.  Comparison of statistical inferences from the DerSimonian–Laird and alternative random‐effects model meta‐analyses – an empirical assessment of 920 Cochrane primary outcome meta‐analyses , 2011, Research synthesis methods.

[41]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[42]  David R. Jones,et al.  How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS , 2005, Statistics in medicine.

[43]  Alexander J Sutton,et al.  What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. , 2004, Statistics in medicine.

[44]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[45]  Heinz Holling,et al.  Nonparametric estimation of heterogeneity variance for the standardised difference used in meta-analysis , 2000 .

[46]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[47]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[48]  Anup Amatya,et al.  Meta-Analysis of Rare Binary Adverse Event Data , 2012, Journal of the American Statistical Association.

[49]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[50]  Wolfgang Viechtbauer,et al.  Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model , 2005 .

[51]  Joseph Beyene,et al.  Ratio of means for analyzing continuous outcomes in meta-analysis performed as well as mean difference methods. , 2011, Journal of clinical epidemiology.

[52]  Guido Knapp,et al.  On Estimating Residual Heterogeneity in Random-Effects Meta-Regression: A Comparative Study , 2013, J. Stat. Theory Appl..

[53]  Kurex Sidik,et al.  Robust variance estimation for random effects meta-analysis , 2006, Comput. Stat. Data Anal..

[54]  Julian P. T. Higgins,et al.  Meta-Regression in Stata , 2008 .

[55]  H Goldstein,et al.  A multilevel model framework for meta-analysis of clinical trials with binary outcomes. , 2000, Statistics in medicine.

[56]  Andrew L. Rukhin,et al.  Estimating heterogeneity variance in meta‐analysis , 2013 .

[57]  Stephen W. Raudenbush,et al.  Analyzing effect sizes: Random-effects models. , 2009 .

[58]  Jack Bowden,et al.  How does the DerSimonian and Laird procedure for random effects meta-analysis compare with its more efficient but harder to compute counterparts? , 2010 .

[59]  Guido Knapp,et al.  Assessing the Amount of Heterogeneity in Random‐Effects Meta‐Analysis , 2006, Biometrical journal. Biometrische Zeitschrift.

[60]  David M. Drukker,et al.  On boundary-value likelihood-ratio tests , 2001 .

[61]  Elena Kulinskaya,et al.  On the moments of Cochran's Q statistic under the null hypothesis, with application to the meta‐analysis of risk difference , 2011, Research synthesis methods.

[62]  I Olkin,et al.  Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. , 2000, Statistics in medicine.

[63]  C. Morris.  Parametric Empirical Bayes Inference: Theory and Applications , 1983 .

[64]  S G Thompson,et al.  A likelihood approach to meta-analysis with random effects. , 1996, Statistics in medicine.

[65]  R. Farebrother.  The Distribution of a Positive Linear Combination of X2 Random Variables , 1984 .

[66]  Evangelos Kontopantelis,et al.  A Re-Analysis of the Cochrane Library Data: The Dangers of Unobserved Heterogeneity in Meta-Analyses , 2013, PloS one.

[67]  Dan Jackson,et al.  The exact distribution of Cochran's heterogeneity statistic in one‐way random effects meta‐analysis , 2008, Statistics in medicine.

[68]  Zhehui Luo,et al.  Fixed effects, random effects and GEE: What are the differences? , 2009, Statistics in medicine.

[69]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[70]  A. Gelman.  Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper) , 2004 .

[71]  Rebecca M. Turner,et al.  Predictive distributions were developed for the extent of heterogeneity in meta-analyses of continuous outcome data , 2015, Journal of clinical epidemiology.

[72]  Sophia Rabe-Hesketh,et al.  A Nondegenerate Penalized Likelihood Estimator for Variance Parameters in Multilevel Models , 2013, Psychometrika.

[73]  John F. Monahan,et al.  Monte Carlo Comparison of ANOVA, MIVQUE, REML, and ML Estimators of Variance Components , 1984 .

[74]  Dan Jackson,et al.  Confidence intervals for the between‐study variance in random effects meta‐analysis using generalised Cochran heterogeneity statistics , 2013, Research synthesis methods.

[75]  Guido Knapp,et al.  Improved tests for a random effects meta‐regression with a single covariate , 2003, Statistics in medicine.

[76]  Elena Kulinskaya,et al.  Testing for Homogeneity in Meta‐Analysis I. The One‐Parameter Case: Standardized Mean Difference , 2009, Biometrics.

[77]  Stephen Senn,et al.  (www.interscience.wiley.com) DOI: 10.1002/sim.2639 Trying to be precise about vagueness , 2022 .

[78]  Douglas G. Altman,et al.  Metan: Fixed- and Random-Effects Meta-Analysis , 2008 .

[79]  Ian R. White,et al.  Multivariate Random-effects Meta-analysis , 2009 .

[80]  Sophia Rabe-Hesketh,et al.  Avoiding zero between‐study variance estimates in random‐effects meta‐analysis , 2013, Statistics in medicine.

[81]  Lili Tian.  Inferences about the between-study variance in meta-analysis with normally distributed outcomes. , 2008, Biometrical journal. Biometrische Zeitschrift.

[82]  F Mosteller,et al.  Some Statistical Methods for Combining Experimental Results , 1990, International Journal of Technology Assessment in Health Care.

[83]  Nikolaos A Patsopoulos,et al.  Uncertainty in heterogeneity estimates in meta-analyses , 2007, BMJ : British Medical Journal.

[84]  D J Spiegelhalter,et al.  Bayesian approaches to random-effects meta-analysis: a comparative study. , 1995, Statistics in medicine.

[85]  C S Berkey,et al.  A random-effects regression model for meta-analysis. , 1995, Statistics in medicine.

[86]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[87]  Evangelos Kontopantelis,et al.  Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation study , 2012, Statistical methods in medical research.

[88]  Kurex Sidik,et al.  Simple heterogeneity variance estimation for meta‐analysis , 2005 .

[89]  Lijuan Wang,et al.  A SAS Interface for Bayesian Analysis With WinBUGS , 2008 .

[90]  Chukiat Viwatwongkasem,et al.  Some general points in estimating heterogeneity variance with the DerSimonian-Laird estimator. , 2002, Biostatistics.

[91]  G. Arnqvist,et al.  MetaWin: Statistical Software for Meta-Analysis with Resampling Tests. Version 1.Michael S. Rosenberg , Dean C. Adams , Jessica Gurevitch , 1998 .

[92]  John Mandel,et al.  Consensus Values and Weighting Factors. , 1982, Journal of research of the National Bureau of Standards.