Publication Bias in Recent Meta-Analyses

Introduction Positive results have a greater chance of being published and outcomes that are statistically significant have a greater chance of being fully reported. One consequence of research underreporting is that it may influence the sample of studies that is available for a meta-analysis. Smaller studies are often characterized by larger effects in published meta-analyses, which can be possibly explained by publication bias. We investigated the association between the statistical significance of the results and the probability of being included in recent meta-analyses. Methods For meta-analyses of clinical trials, we defined the relative risk as the ratio of the probability of including statistically significant results favoring the treatment to the probability of including other results. For meta-analyses of other studies, we defined the relative risk as the ratio of the probability of including biologically plausible statistically significant results to the probability of including other results. We applied a Bayesian selection model for meta-analyses that included at least 30 studies and were published in four major general medical journals (BMJ, JAMA, Lancet, and PLOS Medicine) between 2008 and 2012. Results We identified 49 meta-analyses. The estimate of the relative risk was greater than one in 42 meta-analyses, greater than two in 16 meta-analyses, greater than three in eight meta-analyses, and greater than five in four meta-analyses. In 10 out of 28 meta-analyses of clinical trials, there was strong evidence that statistically significant results favoring the treatment were more likely to be included. In 4 out of 19 meta-analyses of observational studies, there was strong evidence that plausible statistically significant outcomes had a higher probability of being included. Conclusions Publication bias was present in a substantial proportion of large meta-analyses that were recently published in four major medical journals.

[1]  Zhongheng Zhang,et al.  Small studies may overestimate the effect sizes in critical care meta-analyses: a meta-epidemiological study , 2013, Critical Care.

[2]  Theo Vos,et al.  The Long-Term Health Consequences of Child Physical Abuse, Emotional Abuse, and Neglect: A Systematic Review and Meta-Analysis , 2012, PLoS medicine.

[3]  A. Sutton,et al.  Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey , 2012, BMJ : British Medical Journal.

[4]  P. Stang,et al.  Registration of Observational Studies: perspectives from an industry‐based epidemiology group , 2011, Pharmacoepidemiology and drug safety.

[5]  J. Ioannidis,et al.  Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials , 2011, BMJ : British Medical Journal.

[6]  C. Hornuss,et al.  Prevention of pain on injection of propofol: systematic review and meta-analysis , 2011, BMJ : British Medical Journal.

[7]  W. Ghali,et al.  Effect of alcohol consumption on biological markers associated with risk of coronary heart disease: systematic review and meta-analysis of interventional studies , 2011, BMJ : British Medical Journal.

[8]  Kaspar Rufibach,et al.  Selection models with monotone weight functions in meta analysis , 2011, Biometrical journal. Biometrische Zeitschrift.

[9]  W. Ghali,et al.  Association of alcohol consumption with selected cardiovascular disease outcomes: a systematic review and meta-analysis , 2011, BMJ : British Medical Journal.

[10]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[11]  Douglas G Altman,et al.  Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study , 2010, BMJ : British Medical Journal.

[12]  Keith R. Abrams,et al.  Evaluating the Quality of Research into a Single Prognostic Biomarker: A Systematic Review and Meta-analysis of 83 Studies of C-Reactive Protein in Stable Coronary Artery Disease , 2010, PLoS medicine.

[13]  Douglas G Altman,et al.  The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews , 2010, BMJ : British Medical Journal.

[14]  Alex J. Sutton,et al.  Assessing publication bias in meta‐analyses in the presence of between‐study heterogeneity , 2010 .

[15]  Santiago G. Moreno,et al.  Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications , 2009, BMJ : British Medical Journal.

[16]  Seongho Song,et al.  Bayesian meta-analysis using skewed elliptical distributions , 2009 .

[17]  Santiago G. Moreno,et al.  Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study , 2009, BMC medical research methodology.

[18]  Ida Sim,et al.  Publication of Clinical Trials Supporting Successful New Drug Applications: A Literature Analysis , 2008, PLoS medicine.

[19]  J. Ioannidis,et al.  Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias , 2008, PloS one.

[20]  Stephan Morgenthaler,et al.  Meta Analysis: A Guide to Calibrating and Combining Statistical Evidence , 2008 .

[21]  Gerta Rücker,et al.  Arcsine test for publication bias in meta‐analyses with binary outcomes , 2008, Statistics in medicine.

[22]  R. Rosenthal,et al.  Selective publication of antidepressant trials and its influence on apparent efficacy. , 2008, The New England journal of medicine.

[23]  Alex J Sutton,et al.  Performance of the trim and fill method in the presence of publication bias and between‐study heterogeneity , 2007, Statistics in medicine.

[24]  J. Ioannidis,et al.  An exploratory test for an excess of significant findings , 2007, Clinical trials.

[25]  Roger M Harbord,et al.  A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints , 2006, Statistics in medicine.

[26]  Elliot L Jurist.  On Knowing What We Do Not Know , 2006 .

[27]  I. Olkin,et al.  The case of the misleading funnel plot , 2006, BMJ : British Medical Journal.

[28]  P. Gustafson,et al.  Conservative prior distributions for variance parameters in hierarchical models , 2006 .

[29]  Jeffrey D. Kromrey,et al.  On Knowing What We Do Not Know , 2006 .

[30]  A. Sutton,et al.  Comparison of two methods to detect publication bias in meta-analysis. , 2006, JAMA.

[31]  Jerry Nedelman,et al.  Book review: “Bayesian Data Analysis,” Second Edition by A. Gelman, J.B. Carlin, H.S. Stern, and D.B. Rubin Chapman & Hall/CRC, 2004 , 2005, Comput. Stat..

[32]  Jonathan J Deeks,et al.  The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. , 2005, Journal of clinical epidemiology.

[33]  David R. Jones,et al.  How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS , 2005, Statistics in medicine.

[34]  E. Decullier,et al.  Fate of biomedical research protocols and publication bias in France: retrospective cohort study , 2005, BMJ : British Medical Journal.

[35]  A. Gelman.  Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper) , 2004 .

[36]  Ingram Olkin,et al.  Adjusting for publication bias in the presence of heterogeneity , 2003, Statistics in medicine.

[37]  Christian Gluud,et al.  Reported Methodologic Quality and Discrepancies between Large and Small Randomized Trials in Meta-Analyses , 2001, Annals of Internal Medicine.

[38]  J Q Shi,et al.  A sensitivity analysis for publication bias in systematic reviews , 2001, Statistical methods in medical research.

[39]  S D Walter,et al.  A comparison of methods to detect publication bias in meta‐analysis , 2001, Statistics in medicine.

[40]  J. Sterne,et al.  Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. , 2000, Journal of clinical epidemiology.

[41]  S Duval,et al.  Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis , 2000, Biometrics.

[42]  R. Tweedie,et al.  A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis , 2000 .

[43]  G. Smith,et al.  Bias in meta-analysis detected by a simple, graphical test , 1997, BMJ.

[44]  Nancy Paul Silliman,et al.  Hierarchical Selection Models with Applications in Meta-Analysis , 1997 .

[45]  Larry V. Hedges,et al.  Estimating Effect Size Under Publication Bias: Small Sample Properties and Robustness of a Random Effects Selection Model , 1996 .

[46]  C. Begg,et al.  Operating characteristics of a rank correlation test for publication bias. , 1994, Biometrics.

[47]  G. Casella,et al.  Explaining the Gibbs Sampler , 1992 .

[48]  L. Hedges.  Modeling publication selection effects in meta-analysis , 1992 .

[49]  Colin B. Begg,et al.  An Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis , 1992 .

[50]  I Chalmers,et al.  Underreporting research is scientific misconduct. , 1990, JAMA.

[51]  K. Dickersin.  The existence of publication bias and risk factors for its occurrence. , 1990, JAMA.

[52]  Joel B. Greenhouse,et al.  Selection Models and the File Drawer Problem , 1988 .

[53]  G. Rücker,et al.  Treatment-effect estimates adjusted for small-study effects via a limit meta-analysis. , 2011, Biostatistics.

[54]  F. Song,et al.  Bmc Medical Research Methodology Open Access Extent of Publication Bias in Different Categories of Research Cohorts: a Meta-analysis of Empirical Studies , 2009 .

[55]  K R Abrams,et al.  Modelling publication bias in meta-analysis: a review. , 2000, Statistical methods in medical research.

[56]  Bias in meta›analysis detected by a simple, graphical test , 1998 .