A comparison of heterogeneity variance estimators in simulated random‐effects meta‐analyses

Studies combined in a meta-analysis often have differences in their design and conduct that can lead to heterogeneous results. A random-effects model accounts for these differences in the underlying study effects, which includes a heterogeneity variance parameter. The DerSimonian-Laird method is often used to estimate the heterogeneity variance, but simulation studies have found the method can be biased and other methods are available. This paper compares the properties of nine different heterogeneity variance estimators using simulated meta-analysis data. Simulated scenarios include studies of equal size and of moderate and large differences in size. Results confirm that the DerSimonian-Laird estimator is negatively biased in scenarios with small studies and in scenarios with a rare binary outcome. Results also show the Paule-Mandel method has considerable positive bias in meta-analyses with large differences in study size. We recommend the method of restricted maximum likelihood (REML) to estimate the heterogeneity variance over other methods. However, considering that meta-analyses of health studies typically contain few studies, the heterogeneity variance estimate should not be used as a reliable gauge for the extent of heterogeneity in a meta-analysis. The estimated summary effect of the meta-analysis and its confidence interval derived from the Hartung-Knapp-Sidik-Jonkman method are more robust to changes in the heterogeneity variance estimate and show minimal deviation from the nominal coverage of 95% under most of our simulated scenarios.

[1]  George F Borm,et al.  The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method , 2014, BMC Medical Research Methodology.

[2]  Wolfgang Viechtbauer,et al.  Conducting Meta-Analyses in R with the metafor Package , 2010 .

[3]  Joachim Hartung,et al.  Reducing the Number of Unjustified Significant Results in Meta-analysis , 2003 .

[4]  Guido Knapp,et al.  On Estimating Residual Heterogeneity in Random-Effects Meta-Regression: A Comparative Study , 2013, J. Stat. Theory Appl..

[5]  Kit C B Roes,et al.  Estimation of between-trial variance in sequential meta-analyses: a simulation study. , 2014, Contemporary clinical trials.

[6]  Evangelos Kontopantelis,et al.  A Re-Analysis of the Cochrane Library Data: The Dangers of Unobserved Heterogeneity in Meta-Analyses , 2013, PloS one.

[7]  Mark C Simmonds,et al.  A general framework for the use of logistic regression models in meta-analysis , 2016, Statistical methods in medical research.

[8]  Gerta Rücker,et al.  Hartung–Knapp method is not always conservative compared with fixed‐effect meta‐analysis , 2016, Statistics in medicine.

[9]  Theo Stijnen,et al.  Random effects meta‐analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data , 2010, Statistics in medicine.

[10]  Evangelos Kontopantelis,et al.  Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A comparison between DerSimonian–Laird and restricted maximum likelihood , 2012, Statistical methods in medical research.

[11]  Ralf Bender,et al.  Methods to estimate the between‐study variance and its uncertainty in meta‐analysis† , 2015, Research synthesis methods.

[12]  Cynthia D Mulrow,et al.  Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change , 2014, Annals of Internal Medicine.

[13]  David J Spiegelhalter,et al.  Being sceptical about meta-analyses: a Bayesian perspective on magnesium trials in myocardial infarction. , 2002, International journal of epidemiology.

[14]  Theo Stijnen,et al.  A comparison of seven random‐effects models for meta‐analyses that estimate the summary odds ratio , 2018, Statistics in medicine.

[15]  Jack Bowden,et al.  Quantifying, displaying and accounting for heterogeneity in the meta-analysis of RCTs using standard and generalised Q statistics , 2011, BMC medical research methodology.

[16]  Andrew L. Rukhin,et al.  Restricted maximum likelihood estimation of a common mean and the Mandel–Paule algorithm , 2000 .

[17]  Evangelos Kontopantelis,et al.  Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation study , 2012, Statistical methods in medical research.

[18]  J. Hartung,et al.  A refined method for the meta‐analysis of controlled clinical trials with binary outcome , 2001, Statistics in medicine.

[19]  Heinz Holling,et al.  Nonparametric estimation of heterogeneity variance for the standardised difference used in meta-analysis , 2000 .

[20]  Anup Amatya,et al.  Meta-Analysis of Rare Binary Adverse Event Data , 2012, Journal of the American Statistical Association.

[21]  D A Follmann,et al.  Valid Inference in Random Effects Meta‐Analysis , 1999, Biometrics.

[22]  Julian P T Higgins,et al.  An empirical comparison of heterogeneity variance estimators in 12 894 meta‐analyses , 2015, Research synthesis methods.

[23]  Kurex Sidik,et al.  A comparison of heterogeneity variance estimators in combining results of studies , 2007, Statistics in medicine.

[24]  Alexander J Sutton,et al.  What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. , 2004, Statistics in medicine.

[25]  John Mandel,et al.  Consensus Values and Weighting Factors. , 1982, Journal of research of the National Bureau of Standards.

[26]  D. Harville.  Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems , 1977 .

[27]  Andrew L. Rukhin,et al.  Estimating heterogeneity variance in meta‐analysis , 2013 .

[28]  Wolfgang Viechtbauer,et al.  Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model , 2005 .

[29]  Masayuki Henmi,et al.  Confidence intervals for random effects meta‐analysis and robustness to publication bias , 2010, Statistics in medicine.

[30]  Wolfgang Viechtbauer,et al.  Methods for calculating confidence and credible intervals for the residual between-study variance in random effects meta-regression models , 2014, BMC Medical Research Methodology.

[31]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[32]  Julio Sánchez-Meca,et al.  Confidence intervals for the overall effect size in random-effects meta-analysis. , 2008, Psychological methods.

[33]  D J Spiegelhalter,et al.  Bayesian approaches to random-effects meta-analysis: a comparative study. , 1995, Statistics in medicine.

[34]  T. Friede,et al.  Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies , 2015, BMC Medical Research Methodology.

[35]  Guido Knapp,et al.  Improved tests for a random effects meta‐regression with a single covariate , 2003, Statistics in medicine.

[36]  John E. Hunter,et al.  Methods of Meta-Analysis , 1989 .

[37]  Evangelos Kontopantelis,et al.  Metaan: Random-effects Meta-analysis , 2010 .

[38]  Kurex Sidik,et al.  A simple confidence interval for meta‐analysis , 2002, Statistics in medicine.

[39]  C. Morris.  Parametric Empirical Bayes Inference: Theory and Applications , 1983 .

[40]  S G Thompson,et al.  A likelihood approach to meta-analysis with random effects. , 1996, Statistics in medicine.

[41]  Kurex Sidik,et al.  Simple heterogeneity variance estimation for meta‐analysis , 2005 .

[42]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[43]  Jonathan J Deeks,et al.  Much ado about nothing: a comparison of the performance of meta‐analytical methods with rare events , 2007, Statistics in medicine.

[44]  Raghu Kacker,et al.  Random-effects model for meta-analysis of clinical trials: an update. , 2007, Contemporary clinical trials.

[45]  Chukiat Viwatwongkasem,et al.  Some general points in estimating heterogeneity variance with the DerSimonian-Laird estimator. , 2002, Biostatistics.

[46]  Dean Langan,et al.  Comparative performance of heterogeneity variance estimators in meta‐analysis: a review of simulation studies , 2016, Research synthesis methods.

[47]  Rebecca M Turner,et al.  Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive analysis , 2011, BMC medical research methodology.

[48]  Sophia Rabe-Hesketh,et al.  Avoiding zero between‐study variance estimates in random‐effects meta‐analysis , 2013, Statistics in medicine.

[49]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[50]  Mark Simmonds,et al.  A decade of individual participant data meta-analyses: A review of current practice. , 2015, Contemporary clinical trials.