The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method

BackgroundThe DerSimonian and Laird approach (DL) is widely used for random effects meta-analysis, but this often results in inappropriate type I error rates. The method described by Hartung, Knapp, Sidik and Jonkman (HKSJ) is known to perform better when trials of similar size are combined. However evidence in realistic situations, where one trial might be much larger than the other trials, is lacking. We aimed to evaluate the relative performance of the DL and HKSJ methods when studies of different sizes are combined and to develop a simple method to convert DL results to HKSJ results.MethodsWe evaluated the performance of the HKSJ versus DL approach in simulated meta-analyses of 2–20 trials with varying sample sizes and between-study heterogeneity, and allowing trials to have various sizes, e.g. 25% of the trials being 10-times larger than the smaller trials. We also compared the number of “positive” (statistically significant at p < 0.05) findings using empirical data of recent meta-analyses with > = 3 studies of interventions from the Cochrane Database of Systematic Reviews.ResultsThe simulations showed that the HKSJ method consistently resulted in more adequate error rates than the DL method. When the significance level was 5%, the HKSJ error rates at most doubled, whereas for DL they could be over 30%. DL, and, far less so, HKSJ had more inflated error rates when the combined studies had unequal sizes and between-study heterogeneity. The empirical data from 689 meta-analyses showed that 25.1% of the significant findings for the DL method were non-significant with the HKSJ method. DL results can be easily converted into HKSJ results.ConclusionsOur simulations showed that the HKSJ method consistently results in more adequate error rates than the DL method, especially when the number of studies is small, and can easily be applied routinely in meta-analyses. Even with the HKSJ method, extra caution is needed when there are = <5 studies of very unequal sizes.

[1]  Jonathan J Shuster,et al.  Empirical vs natural weighting in random effects meta‐analysis , 2009, Statistics in medicine.

[2]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[3]  B. Djulbegovic,et al.  Allogeneic hematopoietic cell transplantation for adult acute lymphoblastic leukemia (ALL) in first complete remission. , 2011, The Cochrane database of systematic reviews.

[4]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[5]  I Marshall,et al.  Zinc for the common cold. , 1999, The Cochrane database of systematic reviews.

[6]  Masayuki Henmi,et al.  Confidence intervals for random effects meta‐analysis and robustness to publication bias , 2010, Statistics in medicine.

[7]  Kurex Sidik,et al.  Robust variance estimation for random effects meta-analysis , 2006, Comput. Stat. Data Anal..

[8]  J. Ioannidis,et al.  Obtaining evidence by a single well-powered trial or several modestly powered trials , 2016, Statistical methods in medical research.

[9]  Kurex Sidik,et al.  Simple heterogeneity variance estimation for meta‐analysis , 2005 .

[10]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[11]  J. Hartung,et al.  A refined method for the meta‐analysis of controlled clinical trials with binary outcome , 2001, Statistics in medicine.

[12]  D A Follmann,et al.  Valid Inference in Random Effects Meta‐Analysis , 1999, Biometrics.

[13]  Julian P. T. Higgins,et al.  Meta-Regression in Stata , 2008 .

[14]  Kristian Thorlund,et al.  Comparison of statistical inferences from the DerSimonian–Laird and alternative random‐effects model meta‐analyses – an empirical assessment of 920 Cochrane primary outcome meta‐analyses , 2011, Research synthesis methods.

[15]  Kurex Sidik,et al.  On Constructing Confidence Intervals for a Standardized Mean Difference in Meta-analysis , 2003 .

[16]  Joachim Hartung,et al.  An Alternative Method for Meta‐Analysis , 1999 .

[17]  Nikolaos A Patsopoulos,et al.  Uncertainty in heterogeneity estimates in meta-analyses , 2007, BMJ : British Medical Journal.

[18]  Kurex Sidik,et al.  A simple confidence interval for meta‐analysis , 2002, Statistics in medicine.

[19]  Annamaria Guolo,et al.  Higher‐order likelihood inference in meta‐analysis and meta‐regression , 2012, Statistics in medicine.

[20]  S G Thompson,et al.  A likelihood approach to meta-analysis with random effects. , 1996, Statistics in medicine.

[21]  Guido Knapp,et al.  Improved tests for a random effects meta‐regression with a single covariate , 2003, Statistics in medicine.

[22]  Joachim Hartung,et al.  Reducing the Number of Unjustified Significant Results in Meta-analysis , 2003 .

[23]  D. Sackett,et al.  Cochrane Collaboration , 1994, BMJ.

[24]  K. Makambi The Effect of the Heterogeneity Variance Estimator on Some Tests of Treatment Efficacy , 2004, Journal of biopharmaceutical statistics.

[25]  Rebecca M Turner,et al.  Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive analysis , 2011, BMC medical research methodology.

[26]  J. Hartung,et al.  On tests of the overall treatment effect in meta‐analysis with normally distributed responses , 2001, Statistics in medicine.

[27]  Wolfgang Viechtbauer,et al.  Conducting Meta-Analyses in R with the metafor Package , 2010 .

[28]  George F Borm,et al.  The evidence provided by a single trial is less reliable than its statistical analysis suggests. , 2009, Journal of clinical epidemiology.

[29]  Evangelos Kontopantelis,et al.  Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation study , 2012, Statistical methods in medical research.

[30]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[31]  Julio Sánchez-Meca,et al.  Confidence intervals for the overall effect size in random-effects meta-analysis. , 2008, Psychological methods.

[32]  Sarah E Brockwell,et al.  A simple method for inference on an overall effect in meta‐analysis , 2007, Statistics in medicine.