How Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology?

The trustworthiness of research findings has been questioned in many domains of science. This article calls for a review of the trustworthiness of the scientific literature in industrial–organizational (I–O) psychology and a reconsideration of common practices that may harm the credibility of our literature. We note that most hypotheses in I–O psychology journals are confirmed. Thus, we are either approaching omniscience or our journals are publishing an unrepresentative sample of completed research. We view the latter explanation as more likely. We review structural problems in the publication process and in the conduct of research that is likely to promote a distortion of scientific knowledge. We then offer recommendations to make the I–O literature more accurate and trustworthy.

[1]  Kay Dickersin,et al.  Publication Bias: Recognizing the Problem, Understanding Its Origins and Scope, and Preventing Harm , 2006 .

[2]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[3]  Donald P. Schwab,et al.  Research methods for organizational studies, 2nd ed. , 2005 .

[4]  Philip J. Schmeider Stealing into Print: Fraud, Plagiarism, and Misconduct in Scientific Publishing , 1993 .

[5]  J. Ioannidis Why Science Is Not Necessarily Self-Correcting , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[6]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[7]  Wolfgang Viechtbauer,et al.  Publication bias in meta-analysis: Prevention, assessment and adjustments , 2007, Psychometrika.

[8]  Alex J. Sutton,et al.  Evidence Concerning the Consequences of Publication and Related Biases , 2006 .

[9]  T. Sterling Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[10]  Tom Jefferson,et al.  Redundant publication in biomedical sciences: Scientific misconduct or necessity? , 1998, Science and engineering ethics.

[11]  Brian A. Nosek,et al.  An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[12]  D. Rupp Ethical Issues Faced by Editors and Reviewers , 2011, Management and Organization Review.

[13]  ダーウィン チャールス,et al.  The descent of man and selection in relation to sex , 1907 .

[14]  Leif D. Nelson,et al.  Data from Paper “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant” , 2014 .

[15]  Elena L. Grigorenko,et al.  Practical Intelligence in Everyday Life , 2000 .

[16]  Kwok Leung,et al.  Presenting Post Hoc Hypotheses as A Priori: Ethical and Theoretical Issues , 2011, Management and Organization Review.

[17]  J Hilliard,et al.  Again and Again and Again , 2005 .

[18]  E. Reich Plastic Fantastic: How the Biggest Fraud in Physics Shook the Scientific World , 2009 .

[19]  Theodore M. Porter,et al.  Quantification and the Accounting Ideal in Science , 1992 .

[20]  Accessing relevant literature. , 2012 .

[21]  James W. Neuliep,et al.  Editorial bias against replication research. , 1990 .

[22]  M. Shermer,et al.  On Fact and Fraud: Cautionary Tales from the Front Lines of Science , 2010 .

[23]  David B. Balkin,et al.  Determinants of faculty pay: An agency theory perspective. , 1992 .

[24]  John P. A. Ioannidis,et al.  Assessing the Evolution of Effect Sizes over Time , 2006 .

[25]  Michael A. McDaniel,et al.  The Kryptonite of Evidence-Based I–O Psychology , 2011, Industrial and Organizational Psychology.

[26]  J. Wicherts,et al.  Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results , 2011, PloS one.

[27]  Dustin B. Thoman,et al.  Inference Patterns in Theoretical Social Psychology: Looking Back as We Move Forward , 2010 .

[28]  D. Lakens,et al.  Rewarding Replications , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[29]  L. HARKing: Hypothesizing After the Results are Known , 2002 .

[30]  M. Brannick,et al.  How similar are personality scales of the “same” construct? A meta-analytic investigation , 2010 .

[31]  D. Goodstein On Fact and Fraud: Cautionary Tales from the Front Lines of Science , 2010 .

[32]  Sven Kepes,et al.  Meta-analytic Reviews in the Organizational Sciences: Two Meta-analytic Schools on the Way to MARS (the Meta-analytic Reporting Standards) , 2013 .

[33]  Michael A. McDaniel,et al.  Situational judgment test research : Informing the debate on practical intelligence theory , 2005 .

[34]  S. Schmidt Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences , 2009 .

[35]  Arthur G. Bedeian,et al.  Management Science on the Credibility Bubble: Cardinal Sins and Various Misdemeanors , 2010 .

[36]  Sven Kepes,et al.  Avoiding Bias in Publication Bias Research: The Value of “Null” Findings , 2014 .

[37]  Myriam P. Sarachik,et al.  Plastic Fantastic: How the Biggest Fraud in Physics Shook the Scientific World , 2009 .

[38]  T C Chalmers,et al.  Meta-analysis of clinical trials as a scientific discipline. I: Control of bias and comparison with large co-operative trials. , 1987, Statistics in medicine.

[39]  K. Dickersin The existence of publication bias and risk factors for its occurrence. , 1990, JAMA.

[40]  D. Bem Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. , 2011, Journal of personality and social psychology.

[41]  G. Banks,et al.  Publication Bias , 2012 .

[42]  P. Murtaugh,et al.  JOURNAL QUALITY, EFFECT SIZE, AND PUBLICATION BIAS IN META-ANALYSIS , 2002 .

[43]  C. Darwin The Descent of Man and Selection in Relation to Sex: INDEX , 1871 .

[44]  A. Greenwald Consequences of Prejudice Against the Null Hypothesis , 1975 .

[45]  Herman Aguinis,et al.  Revisiting the file drawer problem in meta‐analysis: An assessment of published and nonpublished correlation matrices. , 2012 .

[46]  H. Rothstein,et al.  Revealed or Concealed? Transparency of Procedures, Decisions, and Judgment Calls in Meta-Analyses , 2012 .

[47]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[48]  D. Fanelli Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data , 2010, PloS one.

[49]  M. Hagen,et al.  Ethical principles of psychologists and code of conduct. , 2002, The American psychologist.

[50]  James W. Neuliep,et al.  Reviewer bias against replication research. , 1993 .

[51]  J. Bargh,et al.  Automaticity of social behavior: direct effects of trait construct and stereotype-activation on action. , 1996, Journal of personality and social psychology.

[52]  Jonah Lehrer The Truth Wears Off , 2011 .

[53]  Barry M. Staw,et al.  What Theory is Not , 1995 .

[54]  Bruce Alberts,et al.  Making Data Maximally Available , 2011, Science.

[55]  Han L. J. van der Maas,et al.  Science Perspectives on Psychological an Agenda for Purely Confirmatory Research on Behalf Of: Association for Psychological Science , 2022 .

[56]  A. Liberati Publication bias and the editorial process. , 1992, JAMA.

[57]  John M. Braxton Perspectives on scholarly misconduct in the sciences , 2001 .

[58]  C. Ferguson,et al.  Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. , 2012, Psychological methods.

[59]  D. Fanelli “Positive” Results Increase Down the Hierarchy of the Sciences , 2010, PloS one.

[60]  M. Schminke Editor's Comments: The Better Angels of our Nature—Ethics and Integrity in the Publishing Process , 2009 .

[61]  M. Borenstein,et al.  Publication Bias in Meta-Analysis , 2006 .

[62]  R. Giner-Sorolla,et al.  Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[63]  D. Hambrick THE FIELD OF MANAGEMENT'S DEVOTION TO THEORY: TOO MUCH OF A GOOD THING? , 2007 .

[64]  S. Krimsky Rescuing Science from Politics: Publication Bias, Data Ownership, and the Funding Effect in Science: Threats to the Integrity of Biomedical Research , 2006 .

[65]  R. L. Park,et al.  Voodoo Science: The Road from Foolishness to Fraud , 2000 .

[66]  Cr Sridhar,et al.  Bad Pharma: How drug companies mislead doctors and harm patients , 2013, Journal of Cancer Research and Therapeutics.

[67]  Gregory Francis,et al.  Too good to be true: Publication bias in two prominent studies from experimental psychology , 2012, Psychonomic Bulletin & Review.

[68]  Axel Cleeremans,et al.  Behavioral Priming: It's All in the Mind, but Whose Mind? , 2012, PloS one.

[69]  Jesse A. Berlin,et al.  Preventing Publication Bias: Registries and Prospective Meta‐Analysis , 2006 .

[70]  Michael A. McDaniel,et al.  PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS , 2006 .

[71]  S. Axelrad Field Theory in Social Science: Selected Theoretical Papers by Kurt Lewin , 1951 .

[72]  N. Kerr HARKing: Hypothesizing After the Results are Known , 1998, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[73]  E. Turner,et al.  A Taxpayer-Funded Clinical Trials Registry and Results Database , 2004, PLoS medicine.

[74]  Norman Kaplan,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1974 .

[75]  B. Jasny,et al.  Again, and Again, and Again … , 2011 .

[76]  James R Murphy,et al.  Statistical errors in immunologic research. , 2004, The Journal of allergy and clinical immunology.

[77]  Jeffrey N. Rouder,et al.  A Bayes factor meta-analysis of Bem’s ESP claim , 2011, Psychonomic bulletin & review.

[78]  P. Sleight,et al.  Publication bias , 1991, The Lancet.

[79]  J. Schooler,et al.  Verbal overshadowing of visual memories: Some things are better left unsaid , 1990, Cognitive Psychology.

[80]  D. Harrison,et al.  What's the difference? Diversity constructs as separation, variety, or disparity in organizations. , 2007 .

[81]  Stuart J. Ritchie,et al.  Failing the Future: Three Unsuccessful Attempts to Replicate Bem's ‘Retroactive Facilitation of Recall’ Effect , 2012, PloS one.

[82]  Marc Orlitzky,et al.  How Can Significance Tests Be Deinstitutionalized? , 2012 .

[83]  Donald P. Schwab,et al.  Research Methods for Organizational Studies , 1998 .

[84]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[85]  Apa Publications,et al.  Reporting standards for research in psychology: why do we need them? What might they be? , 2008, The American psychologist.

[86]  R. Nickerson Confirmation Bias: A Ubiquitous Phenomenon in Many Guises , 1998 .

[87]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[88]  J. Ioannidis Why Most Published Research Findings Are False , 2005, PLoS medicine.

[89]  Steve McDonald,et al.  Development of the Cochrane Collaboration’s Central Register of Controlled Clinical Trials , 2002, Evaluation & the health professions.

[90]  F. Rauscher,et al.  Music and spatial task performance , 1993, Nature.

[91]  R. Rosenthal The file drawer problem and tolerance for null results , 1979 .

[92]  Michael A. McDaniel,et al.  Publication Bias in the Organizational Sciences , 2012 .

[93]  D. Borsboom,et al.  The poor availability of psychological research data for reanalysis. , 2006, The American psychologist.

[94]  D. Sarewitz Beware the creeping cracks of bias , 2012, Nature.

[95]  Etienne P LeBel,et al.  Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice , 2011, Review of General Psychology.

[96]  Jeffrey R. Spies,et al.  Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability , 2012, 1205.4251.

[97]  Robert Tibshirani,et al.  Scientific research in the age of omics: the good, the bad, and the sloppy , 2013, J. Am. Medical Informatics Assoc..

[98]  C. Ball,et al.  Repeatability of published microarray gene expression analyses , 2009, Nature Genetics.

[99]  S. Fiedler,et al.  Psychologists Are Open to Change, yet Wary of Rules , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[100]  Thomas A Sellers,et al.  Null Results in Brief: Meeting a Need in Changing Times , 2009, Cancer Epidemiology, Biomarkers & Prevention.

[101]  H. Pashler,et al.  Is the Replicability Crisis Overblown? Three Arguments Examined , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[102]  Daniele Fanelli,et al.  Negative results are disappearing from most disciplines and countries , 2011, Scientometrics.

[103]  J. Ioannidis,et al.  Unavailability of online supplementary scientific information from articles published in major journals , 2005, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[104]  Herman Aguinis,et al.  Meta-Analytic Choices and Judgment Calls: Implications for Theory Building and Testing, Obtained Effect Sizes, and Scholarly Impact , 2011 .

[105]  P. Easterbrook,et al.  Publication bias in clinical research , 1991, The Lancet.

[106]  Michael C. Sturman,et al.  Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes , 2007 .

[107]  Stefan Van Dongen,et al.  Human fluctuating asymmetry in relation to health and quality: a meta-analysis , 2011 .

[108]  Theodor D. Sterling,et al.  Publication decisions revisited: the effect of the outcome of statistical tests on the decision to p , 1995 .

[109]  K. Baggerly Disclose all data in publications. , 2010, Nature.

[110]  P. Shields,et al.  Publication bias is a scientific problem with adverse ethical outcomes: the case for a section for null results. , 2000, Cancer epidemiology, biomarkers & prevention : a publication of the American Association for Cancer Research, cosponsored by the American Society of Preventive Oncology.

[111]  John P. A. Ioannidis,et al.  Concentration of the Most-Cited Papers in the Scientific Literature: Analysis of Journal Ecosystems , 2006, PloS one.

[112]  P. Allison Measures of Inequality , 1978 .

[113]  S. Kerr On the folly of rewarding A, while hoping for B. , 1975, Academy of Management journal. Academy of Management.

[114]  Leroy Wolins,et al.  Responsibility for Raw Data. , 1962 .

[115]  M. Schminke,et al.  RETRACTED - Ethics and Integrity in the Publishing Process: Myths, Facts, and a Roadmap , 2011, Management and Organization Review.

[116]  Sven Kepes,et al.  Publication Bias:A call for improved meta-analytic practice in the organizational sciences , 2012 .

[117]  S. Fiedler,et al.  Is there evidence of publication biases in JDM research? , 2011, Judgment and Decision Making.

[118]  Karl P. Pfeiffer,et al.  The Use of Statistics in Medical Research , 2007 .

[119]  A. Greenwald PSS 7210 . 1177 / 1745691611434210 GreenwaldMethod – Theory Synergy There Is Nothing So Theoretical as a Good Method , 2012 .

[120]  Diane Crawford,et al.  Editorial , 2000, CACM.

[121]  J. Crocker,et al.  Addressing Scientific Fraud , 2011, Science.

[122]  W. Stroebe,et al.  Scientific Misconduct and the Myth of Self-Correction in Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[123]  Ana Marusic,et al.  Clinical trial registration: looking back and moving ahead , 2007, The Lancet.

[124]  G. Vogel Scientific misconduct. Psychologist accused of fraud on 'astonishing scale'. , 2011, Science.