HARKing: How Badly Can Cherry-Picking and Question Trolling Produce Bias in Published Results?

The practice of hypothesizing after results are known (HARKing) has been identified as a potential threat to the credibility of research results. We conducted simulations using input values based on comprehensive meta-analyses and reviews in applied psychology and management (e.g., strategic management studies) to determine the extent to which two forms of HARKing behaviors might plausibly bias study outcomes and to examine the determinants of the size of this effect. When HARKing involves cherry-picking, which consists of searching through data involving alternative measures or samples to find the results that offer the strongest possible support for a particular hypothesis or research question, HARKing has only a small effect on estimates of the population effect size. When HARKing involves question trolling, which consists of searching through data involving several different constructs, measures of those constructs, interventions, or relationships to find seemingly notable results worth writing about, HARKing produces substantial upward bias particularly when it is prevalent and there are many effects from which to choose. Results identify the precise circumstances under which different forms of HARKing behaviors are more or less likely to have a substantial impact on a study’s substantive conclusions and the field’s cumulative knowledge. We offer suggestions for authors, consumers of research, and reviewers and editors on how to understand, minimize, detect, and deter detrimental forms of HARKing in future research.

[1]  Herman Aguinis,et al.  Using Theory Elaboration to Make Theoretical Advancements , 2017 .

[2]  G. Banks,et al.  The Chrysalis Effect , 2017 .

[3]  Marc Orlitzky,et al.  How Can Significance Tests Be Deinstitutionalized? , 2012 .

[4]  Karen Locke,et al.  Perspective - Making Doubt Generative: Rethinking the Role of Doubt in the Research Process , 2008, Organ. Sci..

[5]  Neuroskeptic.  The Nine Circles of Scientific Hell , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[6]  Massimo Pigliucci,et al.  The end of theory in science? , 2009, EMBO reports.

[7]  Herman Aguinis,et al.  What You See is What You Get? Enhancing Methodological Transparency in Management Research , 2017 .

[8]  L. A. Kolesinskaia,et al.  [From the practice]. , 1967, Laboratornoe delo.

[9]  David J. Ketchen,et al.  Research Methodology in Strategic Management , 2008 .

[10]  Peter Lipton,et al.  Testing Hypotheses: Prediction and Prejudice , 2005, Science.

[11]  J. Lampel,et al.  Ethics in the Production and Dissemination of Management Research: Institutional Failure or Individual Fallibility? , 2014 .

[12]  J. Scott Armstrong,et al.  Publication Bias against Null Results , 1997 .

[13]  John C. Scott,et al.  A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology , 2017, Industrial and Organizational Psychology.

[14]  Soon Ang,et al.  The Quantitative Discovery: What is it and How to Get it Published , 2016 .

[15]  A. Jensen,et al.  Bias in Mental Testing. , 1981 .

[16]  E. A. Locke.  The Case for Inductive Theory Building† , 2007 .

[17]  Robert J. Vandenberg,et al.  An Ounce of Prevention Is Worth a Pound of Cure: Improving Research Quality Before Data Collection , 2014 .

[18]  Andrew B. Collmus,et al.  A primer on theory-driven web scraping: Automatic extraction of big data from the Internet for use in psychological research. , 2016, Psychological methods.

[19]  D. Hambrick.  THE FIELD OF MANAGEMENT'S DEVOTION TO THEORY: TOO MUCH OF A GOOD THING? , 2007 .

[20]  Tapabrata Maiti,et al.  Principles and Practice of Structural Equation Modeling (2nd ed.) , 2006 .

[21]  Rachel M. Wasserman,et al.  Ethical Issues and Guidelines for Conducting Data Analysis in Psychological Research , 2013 .

[22]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[23]  R. Vandenberg,et al.  On the Practice of Allowing Correlated Residuals Among Indicators in Structural Equation Models , 2010 .

[24]  Herman Aguinis,et al.  Meta-Analytic Choices and Judgment Calls: Implications for Theory Building and Testing, Obtained Effect Sizes, and Scholarly Impact , 2011 .

[25]  R. Landis,et al.  Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly , 2016 .

[26]  G. Loewenstein,et al.  Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling , 2012, Psychological science.

[27]  John R. Hollenbeck,et al.  Harking, Sharking, and Tharking , 2017 .

[28]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[29]  R. Landis,et al.  When small effect sizes tell a big story, and when large effect sizes don't. , 2009 .

[30]  J. W. Dunlap,et al.  The Vectors of the Mind , 1937 .

[31]  Jeremy B. Bernerth,et al.  A Critical Review and Best‐Practice Recommendations for Control Variable Usage , 2016 .

[32]  Winny Shen,et al.  Samples in applied psychology: over a decade of research in review. , 2011, The Journal of applied psychology.

[33]  Kenneth A. Bollen,et al.  Structural Equations with Latent Variables , 1989 .

[34]  A. Lo,et al.  Data-Snooping Biases in Tests of Financial Asset Pricing Models , 1989 .

[35]  Leslie A. Hayduk.  Structural equation modeling with LISREL: essentials and advances , 1987 .

[36]  D. Fanelli.  How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data , 2009, PloS one.

[37]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[38]  Scott Tonidandel,et al.  Big data at work : the data science revolution and organizational psychology , 2016 .

[39]  D. Sörbom.  Model modification , 1989 .

[40]  L. Thurstone.  The Vectors of Mind , 1935 .

[41]  Herman Aguinis,et al.  HARKing's Threat to Organizational Research: Evidence From Primary and Meta‐Analytic Sources , 2016 .

[42]  Jeffrey Pfeffer,et al.  A Modest Proposal: How We might change the Process and Product of Managerial Research , 2007 .

[43]  Arthur G. Bedeian,et al.  Management Science on the Credibility Bubble: Cardinal Sins and Various Misdemeanors , 2010 .

[44]  W. N. Street,et al.  Financial Asset-Pricing Theory and Stochastic Programming Models for Asset/ Liability Management: a Synthesis , 1996 .

[45]  Thomas G. Cummings,et al.  Scholarly Impact: A Pluralist Conceptualization , 2014 .

[46]  H. Keselman,et al.  Backward, forward and stepwise automated subset selection algorithms: Frequency of obtaining authentic and noise variables , 1992 .

[47]  Constance E. Helfat,et al.  Creating repeatable cumulative knowledge in strategic management , 2016 .

[48]  J. Edwards,et al.  The Presence of Something or the Absence of Nothing: Increasing Theoretical Precision in Management Research , 2010 .

[49]  G. Vining,et al.  Data Analysis: A Model-Comparison Approach , 1989 .

[50]  P. Wright.  Ensuring Research Integrity , 2016 .

[51]  J. M. Cortina,et al.  Twilight of Dawn or of Evening? A Century of Research Methods in the Journal of Applied Psychology , 2017, The Journal of applied psychology.

[52]  Kwok Leung,et al.  Presenting Post Hoc Hypotheses as A Priori: Ethical and Theoretical Issues , 2011, Management and Organization Review.

[53]  Leonard D. Goodstein.  Editorial statement of the American Psychologist: III. , 1986 .

[54]  W. Cascio,et al.  Science’s reproducibility and replicability crisis: International business is not immune , 2017, Journal of International Business Studies.

[55]  David J. Ketchen,et al.  The Use of Archival Proxies in Strategic Management Studies , 2013 .

[56]  J. Shaw.  Advantages of Starting with Theory , 2017 .

[57]  James G. Field,et al.  Correlational effect size benchmarks. , 2015, The Journal of applied psychology.

[58]  Hilda Wing.  STATISTICAL HAZARDS IN THE DETERMINATION OF ADVERSE IMPACT WITH SMALL SAMPLES , 1982 .

[59]  Christopher Hitchcock,et al.  Prediction Versus Accommodation and the Risk of Overfitting , 2004, The British Journal for the Philosophy of Science.

[60]  Herman Aguinis,et al.  Customer-Centric Science: Reporting Significant Research Results With Rigor, Relevance, and Practical Impact in Mind , 2010 .

[61]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[62]  Cheryl L. Adkins,et al.  Questions About Questionable Research Practices in the Field of Management , 2016 .

[63]  Herman Aguinis,et al.  Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings , 2017 .

[64]  Sunil J Rao,et al.  Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis , 2003 .

[65]  N. Kerr.  HARKing: Hypothesizing After the Results are Known , 1998, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[66]  Roger White.  The Epistemic Advantage of Prediction over Accommodation , 2003 .

[67]  D. Ketchen,et al.  Using meta-analytic structural equation modeling to advance strategic management research: Guidelines and an empirical illustration via the strategic leadership-performance relationship , 2016 .