An Assessment of the Magnitude of Effect Sizes

This study compiles information from more than 250 meta-analyses conducted over the past 30 years to assess the magnitude of reported effect sizes in the organizational behavior (OB)/human resources (HR) literatures. Our analysis revealed an average uncorrected effect of r = .227 and an average corrected effect of ρ = .278 (SDρ = .140). Based on the distribution of effect sizes we report, Cohen’s effect size benchmarks are not appropriate for use in OB/HR research as they overestimate the actual breakpoints between small, medium, and large effects. We also assessed the average statistical power reported in meta-analytic conclusions and found substantial evidence that the majority of primary studies in the management literature are statistically underpowered. Finally, we investigated the impact of the file drawer problem in meta-analyses and our findings indicate that the file drawer problem is not a significant concern for meta-analysts. We conclude by discussing various implications of this study for OB/HR researchers.

[1]  Roberto A. Weber,et al.  The Illusion of Leadership: Misattribution of Cause in Coordination Games , 2001 .

[2]  Herman Aguinis,et al.  Revisiting the file drawer problem in meta‐analysis: An assessment of published and nonpublished correlation matrices. , 2012 .

[3]  Sanford B. Ehrlich,et al.  Small Groups: Key Readings , 2006 .

[4]  Donald B. Rubin,et al.  A Note on Percent Variance Explained as A Measure of the Importance of Effects , 1979 .

[5]  N. Roese,et al.  Applications of meta-analysis: 1987-1992 , 1994 .

[6]  T. Judge,et al.  The job satisfaction-job performance relationship: a qualitative and quantitative review. , 2001, Psychological bulletin.

[7]  G. Gigerenzer,et al.  Do studies of statistical power have an effect on the power of studies , 1989 .

[8]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[9]  John E. Penick,et al.  The power of statistical tests in science teachnig research , 1972 .

[10]  C. Ferguson,et al.  Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. , 2012, Psychological methods.

[11]  Hannah R Rothstein,et al.  Publication bias in psychological science: comment on Ferguson and Brannick (2012). , 2012, Psychological methods.

[12]  R. Stogdill.  Personal factors associated with leadership; a survey of the literature. , 1948, The Journal of psychology.

[13]  R. Rosenthal.  Parametric measures of effect size. , 1994 .

[14]  W. W. Rozeboom.  The fallacy of the null-hypothesis significance test. , 1960, Psychological bulletin.

[15]  David J. Ketchen,et al.  Organizational configurations and performance: the role of statistical power in extant research , 1999 .

[16]  Jacob Cohen,et al.  The statistical power of abnormal-social psychological research: a review. , 1962, Journal of abnormal and social psychology.

[17]  L. J. Chase,et al.  A power‐analytic examination of contemporary communication research , 1975 .

[18]  Ralf Schulze,et al.  Meta-Analysis: A Comparison of Approaches , 2004 .

[19]  R. Kirk.  Practical Significance: A Concept Whose Time Has Come , 1996 .

[20]  C. Ferguson.  An effect size primer: A guide for clinicians and researchers. , 2009 .

[21]  Frank Yates,et al.  The Influence of Statistical Methods for Research Workers on the Development of the Science of Statistics , 1951 .

[22]  J. Wicherts,et al.  The Rules of the Game Called Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[23]  Donald B. Rubin,et al.  A Simple, General Purpose Display of Magnitude of Experimental Effect , 1982 .

[24]  M. Borenstein,et al.  Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments , 2006 .

[25]  G. Glass,et al.  Meta-analysis in social research , 1981 .

[26]  F. F. Stephan,et al.  Statistical Procedures and their Mathematical Bases. , 1936 .

[27]  Scott W. Geiger,et al.  Statistical Power and the Testing of Null Hypotheses: A Review of Contemporary Management Research and Recommendations for Future Studies , 2004 .

[28]  Jeffrey C. Kohles,et al.  When the Romance is Over: Follower Perspectives of Aversive Leadership , 2007 .

[29]  Christopher J. Carpenter,et al.  Sample Sizes and Effect Sizes are Negatively Correlated in Meta-Analyses: Evidence and Implications of a Publication Bias Against NonSignificant Findings , 2009 .

[30]  Brian A. Nosek,et al.  Recommendations for Increasing Replicability in Psychology † , 2013 .

[31]  A. Webber,et al.  Publication bias in the medical literature: A review by a Canadian research ethics board , 2007, Canadian journal of anaesthesia = Journal canadien d'anesthesie.

[32]  Herman Aguinis,et al.  Meta-Analytic Choices and Judgment Calls: Implications for Theory Building and Testing, Obtained Effect Sizes, and Scholarly Impact , 2011 .

[33]  T. Scandura,et al.  Research Methodology In Management: Current Practices, Trends, And Implications For Future Research , 2000 .

[34]  R. Arvey,et al.  Meta-Analysis and the Development of Knowledge , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[35]  Irving M. Lane,et al.  A Review of Meta-Analyses in Organizational Behavior and Human Resources Management: An Empirical Assessment , 1991 .

[36]  Michael T Brannick,et al.  Comparison of two random-effects methods of meta-analysis. , 2002, The Journal of applied psychology.

[37]  C. Ferguson,et al.  A Vast Graveyard of Undead Theories , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[38]  Michael A. Campion,et al.  ARTICLE REVIEW CHECKLIST: A CRITERION CHECKLIST FOR REVIEWING RESEARCH ARTICLES IN APPLIED PSYCHOLOGY , 1993 .

[39]  Alan G. Sawyer,et al.  Statistical Power and Effect Size in Marketing Research , 1981 .

[40]  Jacob Cohen,et al.  THINGS I HAVE LEARNED (SO FAR) , 1990 .

[41]  F. Schmidt.  Meta-Analysis , 2008 .

[42]  Mark W. Lipsey,et al.  The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. , 1993, The American psychologist.

[43]  S. Maxwell,et al.  The proof of the pudding: an illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. , 2000, Psychological methods.

[44]  L. J. Chase,et al.  A statistical power analysis of applied psychological research. , 1976 .

[45]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[46]  C. F. Bond,et al.  One Hundred Years of Social Psychology Quantitatively Described , 2003 .

[47]  W. O. Jenkins.  A review of leadership studies with particular reference to military problems. , 1947, Psychological bulletin.

[48]  H. L. Smith,et al.  A brief summary of literature on leadership , 1933 .

[49]  Masoud Hemmasi,et al.  Assessment of statistical power in contemporary strategy research , 1987 .

[50]  James C. Beaty,et al.  Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. , 2005, The Journal of applied psychology.

[51]  Ian Harvey,et al.  Extent of publication bias in different categories of research cohorts: a meta-analysis of empirical studies , 2009, BMC medical research methodology.

[52]  S. West,et al.  Multiple Regression: Testing and Interpreting Interactions. , 1994 .

[53]  C. Argyris.  Personality and organization : the conflict between system and the individual , 1958 .

[54]  James G. Field,et al.  Correlational effect size benchmarks. , 2015, The Journal of applied psychology.

[55]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[56]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[57]  M. Borenstein,et al.  Publication Bias in Meta-Analysis , 2006 .

[58]  A. Greenwald.  Consequences of Prejudice Against the Null Hypothesis , 1975 .

[59]  James K. Brewer,et al.  On the Power of Statistical Tests in the American Educational Research Journal 1 , 1972 .

[60]  Deborah A. Prentice,et al.  When small effects are impressive , 1992 .

[61]  S. Zedeck.  Applied Psychology: Editorial. , 2003 .

[62]  A. Kühberger,et al.  A comprehensive review of reporting practices in psychological journals: Are effect sizes really enough? , 2013 .

[63]  John E. Hunter,et al.  Applications of meta-analysis. , 1987 .

[64]  Piers Steel,et al.  Bayesian Variance Estimation for Meta-Analysis , 2008 .

[65]  B. Bass.  The leaderless group discussion. , 1954, Psychological bulletin.

[66]  L. J. Chase,et al.  Communication disorders: a power analytic assessment of recent research. , 1975, Journal of communication disorders.

[67]  Jeffrey R. Edwards,et al.  To prosper, organizational psychology should … overcome methodological barriers to progress , 2008 .

[68]  Jonathan L. Johnson,et al.  Number of Directors and Financial Performance: A Meta-Analysis , 1999 .

[69]  A. Kluger,et al.  The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. , 1996 .

[70]  Michael A. McDaniel,et al.  Publication Bias in the Organizational Sciences , 2012 .

[71]  S. Maxwell.  The persistence of underpowered studies in psychological research: causes, consequences, and remedies. , 2004, Psychological methods.

[72]  M. Lipsey.  Design sensitivity: Statistical power for applied experimental research. , 1998 .

[73]  G. Glass,et al.  Measures of Association in Comparative Experiments: Their Development and Interpretation , 1969 .

[74]  Mark A. Mone,et al.  THE PERCEPTIONS AND USAGE OF STATISTICAL POWER IN APPLIED PSYCHOLOGY AND MANAGEMENT RESEARCH , 1996 .

[75]  Herman Aguinis,et al.  Customer-Centric Science: Reporting Significant Research Results With Rigor, Relevance, and Practical Impact in Mind , 2010 .

[76]  J. Rossi,et al.  Statistical power of psychological research: what have we gained in 20 years? , 1990, Journal of consulting and clinical psychology.

[77]  R. Fisher.  Statistical methods for research workers , 1927, Protoplasma.

[78]  Masoud Hemmasi,et al.  Statistical Power in Contemporary Management Research , 1987 .

[79]  Samuel B. Bacharach,et al.  Organizational Theories: Some Criteria for Evaluation , 1989 .