Publication Bias:A call for improved meta-analytic practice in the organizational sciences

Previous research has introduced the threat of publication bias to meta-analytic reviews in management and industrial/organizational (I/O) psychology research. However, a comprehensive review of top-tier journals demonstrates that more than two thirds of meta-analytic studies in management and I/O psychology ignore the issue. Of the studies that do empirically evaluate publication bias, almost all use methods that are based on problematic assumptions (e.g., the Failsafe N and subgroup comparisons by source of data).The current paper reviews the issue of publication bias and introduces to management and I/O psychology new methodological techniques to assess this bias.To illustrate the methods, multiple publication bias methods are demonstrated in a meta-analytic review of conditional reasoning tests for aggression.We offer specific recommendations that address both design and analysis issues to mitigate the existence and influence of publication bias.

[1]  Paul R. Sackett,et al.  A META-ANALYSIS OF CONDITIONAL REASONING TESTS OF AGGRESSION , 2010 .

[2]  Michael A. McDaniel,et al.  The Kryptonite of Evidence-Based I–O Psychology , 2011, Industrial and Organizational Psychology.

[3]  Michele Tarsilla Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[4]  A. Field Is the meta-analysis of correlation coefficients accurate when population correlations vary? , 2005, Psychological methods.

[5]  John P. A. Ioannidis,et al.  Assessing the Evolution of Effect Sizes over Time , 2006 .

[6]  Adam R Hafdahl,et al.  Random-effects meta-analysis of correlations: Monte Carlo evaluation of mean estimators. , 2010, The British journal of mathematical and statistical psychology.

[7]  Kay Dickersin,et al.  Publication Bias: Recognizing the Problem, Understanding Its Origins and Scope, and Preventing Harm , 2006 .

[8]  Nicola J Cooper,et al.  Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications , 2009, BMJ : British Medical Journal.

[9]  Lawrence R. James,et al.  Measurement of Personality via Conditional Reasoning , 1998 .

[10]  R. Persaud Misleading meta-analysis , 1996 .

[11]  P. Sleight,et al.  Publication bias , 1991, The Lancet.

[12]  T C Chalmers,et al.  Cumulative meta-analysis of therapeutic trials for myocardial infarction. , 1992, The New England journal of medicine.

[13]  M. Borenstein,et al.  Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments , 2006 .

[14]  Herman Aguinis,et al.  First Decade of Organizational Research Methods , 2009 .

[15]  Sally Hopewell,et al.  Grey Literature and Systematic Reviews , 2006 .

[16]  Murray S. Davis,et al.  That's Interesting! , 1971 .

[17]  J. Ioannidis Differentiating Biases from Genuine Heterogeneity: Distinguishing Artifactual from Substantive Effects , 2006 .

[18]  C. Ferguson,et al.  Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. , 2012, Psychological methods.

[19]  Alex J. Sutton,et al.  Evidence Concerning the Consequences of Publication and Related Biases , 2006 .

[20]  S Duval,et al.  Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis , 2000, Biometrics.

[21]  Adrian Furnham Personality At Work , 1992 .

[22]  T. Jick Mixing Qualitative and Quantitative Methods: Triangulation in Action. , 1979 .

[23]  J. Steenkamp,et al.  A Review and Evaluation of Meta-Analysis Practices in Management Research , 2009 .

[24]  Susan M. Stewart,et al.  An integrative typology of personality assessment for aggression: implications for predicting counterproductive workplace behavior. , 2007, The Journal of applied psychology.

[25]  M. D. Dunnette Handbook of Industrial and Organizational Psychology , 2005 .

[26]  R. Tweedie,et al.  A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis , 2000 .

[27]  Adam R Hafdahl,et al.  Meta-analysis of correlations revisited: attempted replication and extension of Field's (2001) simulation studies. , 2009, Psychological methods.

[28]  W. R. Walton Justification of Antisocial Behavior , 2004 .

[29]  R. Simes,et al.  Publication bias: evidence of delayed publication in a cohort study of clinical research projects , 1997, BMJ.

[30]  J. Ioannidis Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. , 1998, JAMA.

[31]  Jonathan A C Sterne,et al.  Regression Methods to Detect Publication and Other Bias in Meta-Analysis , 2006 .

[32]  Betsy Jane Becker,et al.  Failsafe N or File‐Drawer Number , 2006 .

[33]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[34]  Allen I. Huffcutt,et al.  Development of a new outlier statistic for meta-analytic data. , 1995 .

[35]  C H Schmid,et al.  Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. , 1995, Journal of clinical epidemiology.

[36]  A. Greenwald Consequences of Prejudice Against the Null Hypothesis , 1975 .

[37]  Michael A. McDaniel Cumulative Meta-Analysis as a Publication Bias Method , 2009 .

[38]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[39]  Daniel J Beal,et al.  On the bias of Huffcutt and Arthur's (1995) procedure for identifying outliers in the meta-analysis of correlations. , 2002, The Journal of applied psychology.

[40]  C. Begg,et al.  Operating characteristics of a rank correlation test for publication bias. , 1994, Biometrics.

[41]  Brian C. Frost Implicit and Explicit Personality: An Integrative Approach to Predicting Aggressive Behavior in a Field Setting , 2005 .

[42]  Jesse A. Berlin,et al.  Preventing Publication Bias: Registries and Prospective Meta‐Analysis , 2006 .

[43]  Michael A. McDaniel,et al.  PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS , 2006 .

[44]  Adam R Hafdahl,et al.  Improved Fisher z estimators for univariate random-effects meta-analysis of correlations. , 2009, The British journal of mathematical and statistical psychology.

[45]  Lawrence R. James,et al.  Recording lying, cheating, and defiance in an Internet Based Simulated Environment , 2008, Comput. Hum. Behav..

[46]  R. Orwin A fail-safe N for effect size in meta-analysis. , 1983 .

[47]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[48]  R. Rosenthal The file drawer problem and tolerance for null results , 1979 .

[49]  Daniel G. Bachrach,et al.  Scholarly Influence in the Field of Management: A Bibliometric Analysis of the Determinants of University and Author Impact in the Management Literature in the Past Quarter Century , 2008 .

[50]  R. Briner,et al.  Evidence-Based I–O Psychology: Not There Yet , 2011, Industrial and Organizational Psychology.

[51]  J. Salgado,et al.  Personality at Work , 2020, The Cambridge Handbook of Personality Psychology.

[52]  Sue Duval,et al.  The Trim and Fill Method , 2006 .

[53]  Stephen Evans,et al.  Statistician's comment , 1996 .

[54]  Catherine M. Dalton,et al.  Debunking Myths and Urban Legends About Meta-Analysis , 2011 .

[55]  Robert G. Orwin,et al.  A Fail-SafeN for Effect Size in Meta-Analysis , 1983 .

[56]  A. Uitterlinden,et al.  The Association between Common Vitamin D Receptor Gene Variations and Osteoporosis: A Participant-Level Meta-Analysis , 2006, Annals of Internal Medicine.

[57]  J. Ioannidis,et al.  Evolution and Translation of Research Findings: From Bench to Where? , 2006, PLoS clinical trials.

[58]  J. Ioannidis,et al.  Unavailability of online supplementary scientific information from articles published in major journals , 2005, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[59]  Herman Aguinis,et al.  Meta-Analytic Choices and Judgment Calls: Implications for Theory Building and Testing, Obtained Effect Sizes, and Scholarly Impact , 2011 .

[60]  F. Song,et al.  Dissemination and publication of research findings: an updated review of related biases. , 2010, Health technology assessment.

[61]  M. Borenstein,et al.  Publication Bias in Meta-Analysis , 2006 .

[62]  John E. Hunter,et al.  Dichotomization of continuous variables: the implications for meta-analysis , 1990 .

[63]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[64]  James M. LeBreton,et al.  A Conditional Reasoning Measure for Aggression , 2005 .

[65]  A. Field Meta-analysis of correlation coefficients: a Monte Carlo comparison of fixed- and random-effects methods. , 2001, Psychological methods.

[66]  Ingram Olkin,et al.  Adjusting for publication bias in the presence of heterogeneity , 2003, Statistics in medicine.

[67]  Michael A. McDaniel,et al.  A reexamination of black-white mean differences in work performance: more data, more moderators. , 2006, The Journal of applied psychology.

[68]  J. R. Larson,et al.  Research strategies and tactics in industrial and organizational psychology. , 1990 .

[69]  Michael A. McDaniel,et al.  Race Differences in Personality: An Evaluation of Moderators and Publication Bias , 2008 .

[70]  Michael A. McDaniel,et al.  Smarter people are (a bit) more symmetrical: A meta-analysis of the relationship between intelligence and fluctuating asymmetry , 2010 .

[71]  Lois E. Tetrick,et al.  Society for Industrial and Organizational Psychology , 2010 .