The garden of forking paths : Why multiple comparisons can be a problem , even when there is no “ fishing expedition ” or “ p-hacking ” and the research hypothesis was posited ahead of time ∗

Researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data. The problem is there can be a large number of potential comparisons when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of fishing or examining multiple p-values. We discuss in the context of several examples of published papers where data-analysis decisions were theoretically-motivated based on previous literature, but where the details of data selection and analysis were not pre-specified and, as a result, were contingent on data. 1. Multiple comparisons doesn’t have to feel like fishing

[1]  Francisco Cardoso,et al.  Too good to be true , 2018, Arquivos de Neuro-Psiquiatria.

[2]  A. Gelman.  The Connection Between Varying Treatment Effects and the Crisis of Unreplicable Research , 2015 .

[3]  G. Francis.  Replication, statistical consistency, and publication bias. , 2013 .

[4]  Jessica L. Tracy,et al.  Women Are More Likely to Wear Red or Pink at Peak Fertility , 2013, Psychological science.

[5]  John Tooby,et al.  The Ancestral Logic of Politics , 2013, Psychological science.

[6]  Vladas Griskevicius,et al.  The Fluctuating Female Vote , 2013, Psychological science.

[7]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[8]  Sander Greenland,et al.  Living with P Values: Resurrecting a Bayesian Perspective on Frequentist Statistics , 2013, Epidemiology.

[9]  Macartan Humphreys,et al.  Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration , 2012, Political Analysis.

[10]  James E. Monogan.  A Case for Registering Studies of Political Outcomes: An Application in the 2010 House Elections , 2013, Political Analysis.

[11]  Is It Possible to Be an Ethicist Without Being Mean to People ? 1 , 2013 .

[12]  Brian A. Nosek,et al.  Scientific Utopia , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[13]  W. Johnson,et al.  Must psychologists change the way they analyze their data? , 2011, Journal of personality and social psychology.

[14]  E. Wagenmakers,et al.  Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). , 2011, Journal of personality and social psychology.

[15]  D. Bem.  Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. , 2011, Journal of personality and social psychology.

[16]  Michael B. Miller,et al.  of Serendipitous and Unexpected Results Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon : An Argument For Proper Multiple Comparisons Correction , 2010 .

[17]  H. Pashler,et al.  Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition 1 , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[18]  A. Gelman,et al.  Of Beauty, Sex and Power , 2009 .

[19]  S. Mullainathan,et al.  Sticking with Your Vote: Cognitive Dissonance and Political Attitudes , 2009 .

[20]  Andrew Gelman,et al.  Why We (Usually) Don't Have to Worry About Multiple Comparisons , 2009, 0907.2478.

[21]  A. Gelman,et al.  Rich State, Poor State, Red State, Blue State: What's the Matter with Connecticut? , 2005 .

[22]  Andrew Gelman,et al.  Treatment Effects in Before‐After Data , 2005 .

[23]  Francis Tuerlinckx,et al.  Type S error rates for classical and Bayesian single and multiple comparison procedures , 2000, Comput. Stat..

[24]  George E. P. Box.  SCIENTIFIC METHOD : THE GENERATION OF KNOWLEDGE AND QUALITY , 1997 .

[25]  Gary King,et al.  Enhancing Democracy Through Legislative Redistricting , 1994, American Political Science Review.

[26]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[27]  John W. Tukey,et al.  We Need Both Exploratory and Confirmatory , 1980 .

[28]  A. D. de Groot.  [The meaning of the concept of significance in studies of various types]. , 1956, Nederlands tijdschrift voor de psychologie en haar grensgebieden.

[29]  Jorge Luis Borges,et al.  El jardín de senderos que se bifurcan , 1941 .