Avoiding Bias in Publication Bias Research: The Value of “Null” Findings

Meta-analytic reviews are an important tool for advancing science and guiding evidence-based practice. Publication bias is one of the greatest threats to meta-analytic reviews. This paper assesses the degree of publication bias in four previously published meta-analytic datasets from various fields of study in the organizational sciences. Of these datasets, one appears to be relatively unaffected by publication bias while the others seem to be noticeably influenced by this bias. Our “null” result (i.e., a prior meta-analytic estimate is unlikely to have been affected by publication bias) increases our confidence in the accuracy of our cumulative knowledge. Yet, our other findings suggest the presence of publication bias and point to the need for caution and further research.

[1]  Kay Dickersin,et al.  Publication Bias: Recognizing the Problem, Understanding Its Origins and Scope, and Preventing Harm , 2006 .

[2]  Sally Hopewell,et al.  Grey Literature and Systematic Reviews , 2006 .

[3]  Murray S. Davis,et al.  That's Interesting! , 1971 .

[4]  Alex J Sutton,et al.  Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. , 2008, Journal of clinical epidemiology.

[5]  Kate E Decleene,et al.  Publication Manual of the American Psychological Association , 2011 .

[6]  Michael A. McDaniel,et al.  The Kryptonite of Evidence-Based I–O Psychology , 2011, Industrial and Organizational Psychology.

[7]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[8]  J. Ioannidis,et al.  Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials , 2011, BMJ : British Medical Journal.

[9]  Sally Hopewell,et al.  Publication bias in clinical trials due to statistical significance or direction of trial results. , 2009, The Cochrane database of systematic reviews.

[10]  M. V. van Engen,et al.  Transformational, transactional, and laissez-faire leadership styles: a meta-analysis comparing women and men. , 2003, Psychological bulletin.

[11]  Carole Torgerson,et al.  PUBLICATION BIAS: THE ACHILLES’ HEEL OF SYSTEMATIC REVIEWS? , 2006 .

[12]  Alex J. Sutton,et al.  Evidence Concerning the Consequences of Publication and Related Biases , 2006 .

[13]  Sue Duval,et al.  The Trim and Fill Method , 2006 .

[14]  Stephen Evans,et al.  Statistician's comment , 1996 .

[15]  Carol M Woods,et al.  Publication bias in research synthesis: sensitivity analysis using a priori weight functions. , 2005, Psychological methods.

[16]  C. Ferguson,et al.  Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. , 2012, Psychological methods.

[17]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[18]  Catherine M. Dalton,et al.  Debunking Myths and Urban Legends About Meta-Analysis , 2011 .

[19]  G. Banks,et al.  Publication Bias , 2012 .

[20]  S. Blackwell,et al.  Full Publication of Clinical Trials Presented at a National Maternal-Fetal Medicine Meeting: Is There a Publication Bias? , 2009, American journal of perinatology.

[21]  M. Borenstein,et al.  Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments , 2006 .

[22]  J. R. Larson,et al.  Research strategies and tactics in industrial and organizational psychology. , 1990 .

[23]  Theodor D. Sterling,et al.  Publication decisions revisited: the effect of the outcome of statistical tests on the decision to p , 1995 .

[24]  T. Scandura,et al.  Research Methodology In Management: Current Practices, Trends, And Implications For Future Research , 2000 .

[25]  Alex J. Sutton,et al.  Assessing publication bias in meta‐analyses in the presence of between‐study heterogeneity , 2010 .

[26]  F. Song,et al.  Dissemination and publication of research findings: an updated review of related biases. , 2010, Health technology assessment.

[27]  S. Robbins,et al.  Intervention Effects on College Performance and Retention as Mediated by Motivational, Emotional, and Social Control Factors: Integrated Meta-Analytic Path Analyses , 2009, The Journal of applied psychology.

[28]  H. Cooper,et al.  Foundations, planning, measures, and psychometrics , 2012 .

[29]  R. Briner,et al.  Evidence-Based I–O Psychology: Not There Yet , 2011, Industrial and Organizational Psychology.

[30]  Ralf Schulze,et al.  Meta-Analysis: A Comparison of Approaches , 2004 .

[31]  Joachim Schöpfel.  Observations on the Future of Grey Literature , 2006 .

[32]  John P. A. Ioannidis,et al.  Assessing the Evolution of Effect Sizes over Time , 2006 .

[33]  R. Rosenthal,et al.  Selective publication of antidepressant trials and its influence on apparent efficacy. , 2008, The New England journal of medicine.

[34]  Nicola J Cooper,et al.  Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications , 2009, BMJ : British Medical Journal.

[35]  M Egger,et al.  Grey literature in meta-analyses of randomized trials of health care interventions. , 2007, The Cochrane database of systematic reviews.

[36]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[37]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[38]  Accessing relevant literature. , 2012 .

[39]  Joel B. Greenhouse,et al.  Sensitivity analysis and diagnostics. , 1994 .

[40]  D. McNatt,et al.  Ancient Pygmalion joins contemporary management: a meta-analysis of the result. , 2000, The Journal of applied psychology.

[41]  M. Egger,et al.  Who benefits from medical interventions? , 1994, BMJ.

[42]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[43]  Betsy Jane Becker,et al.  Combining significance levels. , 1994 .

[44]  M. D. Dunnette.  Handbook of Industrial and Organizational Psychology , 2005 .

[45]  George Davey Smith,et al.  meta-analysis bias in location and selection of studies , 1998 .

[46]  Harris Cooper,et al.  Psychological Bulletin: Editorial. , 2003 .

[47]  J. Scott Armstrong,et al.  Publication Bias against Null Results , 1997 .

[48]  Huy Le,et al.  Implications of Methodological Advances for the Practice of Personnel Selection: How Practitioners Benefit from Meta-analysis , 2007 .

[49]  Andy P Field,et al.  How to do a meta-analysis. , 2010, The British journal of mathematical and statistical psychology.

[50]  Jeffrey M Drazen,et al.  Expression of concern reaffirmed. , 2006, The New England journal of medicine.

[51]  J. Ioannidis,et al.  Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias , 2008, PloS one.

[52]  P. Mayring.  Mixing Qualitative and Quantitative Methods , 2007 .

[53]  R. Persaud.  Misleading meta-analysis , 1996 .

[54]  Ingram Olkin,et al.  Adjusting for publication bias in the presence of heterogeneity , 2003, Statistics in medicine.

[55]  Jesse A. Berlin,et al.  Preventing Publication Bias: Registries and Prospective Meta‐Analysis , 2006 .

[56]  Michael A. McDaniel,et al.  PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS , 2006 .

[57]  J. Ioannidis,et al.  Unavailability of online supplementary scientific information from articles published in major journals , 2005, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[58]  K. Dickersin.  The existence of publication bias and risk factors for its occurrence. , 1990, JAMA.

[59]  L. Hedges,et al.  The Handbook of Research Synthesis , 1995 .

[60]  Jonathan A C Sterne,et al.  Regression Methods to Detect Publication and Other Bias in Meta-Analysis , 2006 .

[61]  A. Greenwald.  Consequences of Prejudice Against the Null Hypothesis , 1975 .

[62]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[63]  C. Begg,et al.  Operating characteristics of a rank correlation test for publication bias. , 1994, Biometrics.

[64]  Jonathan A C Sterne,et al.  The Funnel Plot , 2006 .

[65]  Dan Remenyi,et al.  Research Strategies and Tactics , 1998 .

[66]  Sven Kepes,et al.  Publication Bias:A call for improved meta-analytic practice in the organizational sciences , 2012 .

[67]  S. Fiedler,et al.  Is there evidence of publication biases in JDM research? , 2011, Judgment and Decision Making.

[68]  T. Jick.  Mixing Qualitative and Quantitative Methods: Triangulation in Action. , 1979 .

[69]  Alex J. Sutton,et al.  Contour-Enhanced Funnel Plots for Meta-Analysis , 2008 .

[70]  L. Hedges.  Modeling publication selection effects in meta-analysis , 1992 .

[71]  Michael T. Brannick,et al.  An Illustration of the Consequences of Meta-Analysis Model Choice , 2008 .

[72]  S Duval,et al.  Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis , 2000, Biometrics.

[73]  Mark S. Teachout,et al.  THE RELATIONSHIP BETWEEN WORK EXPERIENCE AND JOB PERFORMANCE: A CONCEPTUAL AND META‐ANALYTIC REVIEW , 1995 .

[74]  S. Brophy,et al.  Interventions for latent autoimmune diabetes (LADA) in adults. , 2011, The Cochrane database of systematic reviews.

[75]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[76]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[77]  Michael A. McDaniel,et al.  Publication Bias in the Organizational Sciences , 2012 .

[78]  I. Robertson,et al.  The Validity of Assessment Centers for the Prediction of Supervisory Performance Ratings: A Meta-Analysis , 2007 .

[79]  In-Sue Oh,et al.  Fixed- versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results. , 2009, The British journal of mathematical and statistical psychology.

[80]  Michael A. McDaniel,et al.  Smarter people are (a bit) more symmetrical: A meta-analysis of the relationship between intelligence and fluctuating asymmetry , 2010 .

[81]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[82]  R. Tweedie,et al.  A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis , 2000 .

[83]  Andreas Rosenblad,et al.  Introduction to Meta‐Analysis by Michael Borenstein, Larry V. Hedges, Julian P.T. Higgins, Hannah R. Rothstein , 2009 .

[84]  C. Anderson,et al.  Gender differences in aggression as a function of provocation: a meta-analysis. , 1996, Psychological bulletin.