A Re-Analysis of the Cochrane Library Data: The Dangers of Unobserved Heterogeneity in Meta-Analyses

Background Heterogeneity has a key role in meta-analysis methods and can greatly affect conclusions. However, true levels of heterogeneity are unknown and often researchers assume homogeneity. We aim to: a) investigate the prevalence of unobserved heterogeneity and the validity of the assumption of homogeneity; b) assess the performance of various meta-analysis methods; c) apply the findings to published meta-analyses. Methods and Findings We accessed 57,397 meta-analyses, available in the Cochrane Library in August 2012. Using simulated data we assessed the performance of various meta-analysis methods in different scenarios. The prevalence of a zero heterogeneity estimate in the simulated scenarios was compared with that in the Cochrane data, to estimate the degree of unobserved heterogeneity in the latter. We re-analysed all meta-analyses using all methods and assessed the sensitivity of the statistical conclusions. Levels of unobserved heterogeneity in the Cochrane data appeared to be high, especially for small meta-analyses. A bootstrapped version of the DerSimonian-Laird approach performed best in both detecting heterogeneity and in returning more accurate overall effect estimates. Re-analysing all meta-analyses with this new method we found that in cases where heterogeneity had originally been detected but ignored, 17–20% of the statistical conclusions changed. Rates were much lower where the original analysis did not detect heterogeneity or took it into account, between 1% and 3%. Conclusions When evidence for heterogeneity is lacking, standard practice is to assume homogeneity and apply a simpler fixed-effect meta-analysis. We find that assuming homogeneity often results in a misleading analysis, since heterogeneity is very likely present but undetected. Our new method represents a small improvement but the problem largely remains, especially for very small meta-analyses. One solution is to test the sensitivity of the meta-analysis conclusions to assumed moderate and large degrees of heterogeneity. Equally, whenever heterogeneity is detected, it should not be ignored.

[1]  Jonathan J Deeks,et al.  Statistical algorithms in Review Manager , 2010 .

[2]  W. G. Cochran Problems arising in the analysis of a series of similar experiments , 1937 .

[3]  Evangelos Kontopantelis,et al.  Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation study , 2012, Statistical methods in medical research.

[4]  Kurex Sidik,et al.  Simple heterogeneity variance estimation for meta‐analysis , 2005 .

[5]  H. Heinzl,et al.  A simulation study comparing properties of heterogeneity measures in meta‐analyses , 2006, Statistics in medicine.

[6]  Douglas G. Altman,et al.  Systematic Reviews in Health Care , 2001 .

[7]  Kurex Sidik,et al.  A simple confidence interval for meta‐analysis , 2002, Statistics in medicine.

[8]  Simon G Thompson,et al.  Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews , 2012, International journal of epidemiology.

[9]  M. Pittler Systematic Reviews in Health Care: Meta‐analysis in Context , 2010 .

[10]  D A Follmann,et al.  Valid Inference in Random Effects Meta‐Analysis , 1999, Biometrics.

[11]  C. Morris Parametric Empirical Bayes Inference: Theory and Applications , 1983 .

[12]  Larry V. Hedges,et al.  A random effects model for effect sizes , 1983 .

[13]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[14]  Douglas G. Altman,et al.  Statistical Methods for Examining Heterogeneity and Combining Results from Several Studies in Meta‐Analysis , 2008 .

[15]  S G Thompson,et al.  A likelihood approach to meta-analysis with random effects. , 1996, Statistics in medicine.

[16]  Jack Bowden,et al.  How does the DerSimonian and Laird procedure for random effects meta-analysis compare with its more efficient but harder to compute counterparts? , 2010 .

[17]  B J Biggerstaff,et al.  Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. , 1997, Statistics in medicine.

[18]  B. Efron Nonparametric estimates of standard error: The jackknife, the bootstrap and other methods , 1981 .

[19]  Andrew L. Rukhin,et al.  Estimating heterogeneity variance in meta‐analysis , 2013 .

[20]  W. Haenszel,et al.  Statistical aspects of the analysis of data from retrospective studies of disease. , 1959, Journal of the National Cancer Institute.

[21]  Evangelos Kontopantelis,et al.  Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A comparison between DerSimonian–Laird and restricted maximum likelihood , 2012, Statistical methods in medical research.

[22]  Raghu Kacker,et al.  Random-effects model for meta-analysis of clinical trials: an update. , 2007, Contemporary clinical trials.

[23]  W. G. Cochran The combination of estimates from different experiments. , 1954 .

[24]  Deborah H. Charbonneau,et al.  Doody's Core Titles in the Health Sciences (DCT) , 2005 .

[25]  Evangelos Kontopantelis,et al.  Metaan: Random-effects Meta-analysis , 2010 .

[26]  S. Thompson,et al.  Detecting and describing heterogeneity in meta-analysis. , 1998, Statistics in medicine.

[27]  Kurex Sidik,et al.  A comparison of heterogeneity variance estimators in combining results of studies , 2007, Statistics in medicine.

[28]  R. Peto,et al.  Beta blockade during and after myocardial infarction: an overview of the randomized trials. , 1985, Progress in cardiovascular diseases.

[29]  Sarah E. Brockwell,et al.  A comparison of statistical methods for meta‐analysis , 2001, Statistics in medicine.

[30]  J M Robins,et al.  Estimation of a common effect parameter from sparse follow-up data. , 1985, Biometrics.

[31]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.