Assessing publication bias in meta‐analyses in the presence of between‐study heterogeneity

Between-study heterogeneity and publication bias are common features of a meta-analysis that can be present simultaneously. When both are suspected, consideration must be made of each in the assessment of the other. We consider extended funnel plot tests for detecting publication bias, and selection modelling and trim-and-fill methods to adjust for publication bias in the presence of between-study heterogeneity. These methods are applied to two example data sets. Results indicate that ignoring between-study heterogeneity when assessing publication bias can be misleading, but that methods to test or adjust for publication bias in the presence of heterogeneity may not be powerful when the meta-analysis is not large. It is therefore unrealistic to expect to disentangle the effects of publication bias and heterogeneity reliably in all except the largest meta-analyses. Copyright (c) 2010 Royal Statistical Society.

[1]  C. Begg,et al.  Operating characteristics of a rank correlation test for publication bias. , 1994, Biometrics.

[2]  I Olkin,et al.  Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. , 2000, Statistics in medicine.

[3]  David B. Pillemer,et al.  Summing Up: The Science of Reviewing Research , 1984 .

[4]  Santiago G. Moreno,et al.  Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study , 2009, BMC medical research methodology.

[5]  Christopher H Schmid,et al.  In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. , 2005, Journal of clinical epidemiology.

[6]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[7]  Larry V. Hedges,et al.  Estimating Effect Size Under Publication Bias: Small Sample Properties and Robustness of a Random Effects Selection Model , 1996 .

[8]  J. Mapstone,et al.  Fluid resuscitation strategies: a systematic review of animal trials. , 2003, The Journal of trauma.

[9]  Thomas A Trikalinos,et al.  The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey , 2007, Canadian Medical Association Journal.

[10]  S Duval,et al.  Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis , 2000, Biometrics.

[11]  Jonathan J Deeks,et al.  The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. , 2005, Journal of clinical epidemiology.

[12]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[13]  Stephen W. Raudenbush,et al.  Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. , 1984 .

[14]  Alex J. Sutton,et al.  Publication and related biases: a review , 2000 .

[15]  Dan Jackson,et al.  The implications of publication bias for meta‐analysis' other parameter , 2006, Statistics in medicine.

[16]  Guido Schwarzer,et al.  Inflation of type I error rate in two statistical tests for the detection of publication bias in meta‐analyses with binary outcomes , 2002, Statistics in medicine.

[17]  J. Copas.  What works?: selectivity models and meta‐analysis , 1999 .

[18]  Keith R Abrams,et al.  Generalized synthesis of evidence and the threat of dissemination bias. the example of electronic fetal heart rate monitoring (EFM). , 2002, Journal of clinical epidemiology.

[19]  Ingram Olkin,et al.  Adjusting for publication bias in the presence of heterogeneity , 2003, Statistics in medicine.

[20]  Alex J Sutton,et al.  Performance of the trim and fill method in the presence of publication bias and between‐study heterogeneity , 2007, Statistics in medicine.

[21]  R. Tweedie,et al.  A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis , 2000 .

[22]  J. Sterne,et al.  Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. , 2000, Journal of clinical epidemiology.

[23]  Gerta Rücker,et al.  Arcsine test for publication bias in meta‐analyses with binary outcomes , 2008, Statistics in medicine.

[24]  Carol M Woods,et al.  Publication bias in research synthesis: sensitivity analysis using a priori weight functions. , 2005, Psychological methods.

[25]  S D Walter,et al.  A comparison of methods to detect publication bias in meta‐analysis , 2001, Statistics in medicine.

[26]  S. Lewis,et al.  Meta-analysis of MTHFR 677C→ T polymorphism and coronary heart disease: does totality of evidence support causal role for homocysteine and preventive potential of folate? , 2005, BMJ : British Medical Journal.

[27]  A. Sutton,et al.  Comparison of two methods to detect publication bias in meta-analysis. , 2006, JAMA.

[28]  Roger M Harbord,et al.  A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints , 2006, Statistics in medicine.

[29]  G. Smith,et al.  Bias in meta-analysis detected by a simple, graphical test , 1997, BMJ.

[30]  David R. Jones,et al.  Empirical assessment of effect of publication bias on meta-analyses , 2000, BMJ : British Medical Journal.