Special Issue "Strengthening Derivation Chains in Cognitive Neuroscience": Research Report We don ' t know what you did last summer. On the importance of transparent reporting of reaction time data pre-processing

systematic literature review , presented in this paper, revealed that the justification for and order in which analytical choices are conducted are rarely reported, leading to difficulty in reproducing results and interpreting mixed findings. To address this methodological shortcoming, we created a checklist on reporting reaction time pre-processing to make these decisions more explicit, improve transparency, and thus, promote best practices within the field. The importance of the pre-processing checklist was additionally supported by an expert consensus survey and a multiverse analysis . Consequently, we appeal for maximal transparency on all methods applied and offer a checklist to improve replicability and reproducibility of studies that use reaction time measures.

[1]  C. Li,et al.  Differences in Ex-Gaussian Parameters from Response Time Distributions Between Individuals with and Without Attention Deficit/Hyperactivity Disorder: A Meta-analysis , 2023, Neuropsychology review.

[2]  Gijsbert Bijlstra,et al.  The effect of face masks on the stereotype effect in emotion perception , 2022, Journal of Experimental Social Psychology.

[3]  Gijsbert Bijlstra,et al.  Face masks impair facial emotion recognition and induce specific emotion confusions , 2022, Cognitive Research: Principles and Implications.

[4]  Sufyan bin Uzayr.  GitHub , 2022, Mastering Git.

[5]  C. Bokhove.  The role of analytical variability in secondary data replications: a replication of Kim et al. (2014) , 2022, Educational Research and Evaluation.

[6]  Martin R. Vasilev,et al.  Missing Data and Multiple Imputation Decision Tree , 2021 .

[7]  M. Kiefer,et al.  Comparison of Different Response Time Outlier Exclusion Methods: A Simulation Study , 2021, Frontiers in Psychology.

[8]  C. Pollo,et al.  Proactive inhibition is not modified by deep brain stimulation for Parkinson's disease: An electrical neuroimaging study , 2021, Human brain mapping.

[9]  Martin R. Vasilev,et al.  Best Practices for Addressing Missing Data through Multiple Imputation , 2021, Infant and Child Development.

[10]  B. Doolan,et al.  the United Nations Geoscheme , 2021 .

[11]  Quentin André.  Outlier exclusion procedures must be blind to the researcher's hypothesis. , 2021, Journal of experimental psychology. General.

[12]  B. Hommel,et al.  The neurocognitive underpinnings of the Simon effect: An integrative review of current research , 2020, Cognitive, Affective, & Behavioral Neuroscience.

[13]  Ulrich Dirnagl,et al.  The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research* , 2020, BMC Veterinary Research.

[14]  Michèle B. Nuijten,et al.  Replicability, Robustness, and Reproducibility in Psychological Science. , 2020, Annual review of psychology.

[15]  Miguel A. Vadillo,et al.  Flexibility in reaction time analysis: many roads to a false positive? , 2020, Royal Society Open Science.

[16]  Robert T. Thibault,et al.  Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014-2017) , 2020 .

[17]  D. Lakens.  The Value of Preregistration for Psychological Science: A Conceptual Analysis , 2019 .

[18]  Bradley C. Love,et al.  Variability in the analysis of a single neuroimaging dataset by many teams , 2019, Nature.

[19]  E. Paluck,et al.  Open Science Practices are on the Rise: The State of Social Science (3S) Survey , 2019 .

[20]  Mathew H. Evans,et al.  Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results , 2018, Advances in Methods and Practices in Psychological Science.

[21]  A. Ehlis,et al.  Reduction but no shift in brain activation after arithmetic learning in children: A simultaneous fNIRS-EEG study , 2018, Scientific Reports.

[22]  James Allen Kerr,et al.  Choice of analysis pathway dramatically affects statistical outcomes in breaking continuous flash suppression , 2017, Scientific Reports.

[23]  Robbie C. M. van Aert,et al.  Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking , 2016, Front. Psychol..

[24]  A. Gelman,et al.  Increasing Transparency Through a Multiverse Analysis , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[25]  Lex Nederbragt,et al.  Good enough practices in scientific computing , 2016, PLoS Comput. Biol..

[26]  M. Behrmann,et al.  Attentional dynamics mediated by subcortical mechanisms , 2014, Attention, Perception, & Psychophysics.

[27]  B. Burle,et al.  Distributional reaction time properties in the Eriksen task: marked differences or hidden similarities with the Simon task? , 2013, Psychonomic bulletin & review.

[28]  Daniël Lakens,et al.  Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs , 2013, Front. Psychol..

[29]  Christophe Ley,et al.  Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median , 2013 .

[30]  Michael J Cortese,et al.  The semantic priming project , 2013, Behavior research methods.

[31]  Herman Aguinis,et al.  Best-Practice Recommendations for Defining, Identifying, and Handling Outliers , 2013 .

[32]  Jordan Litman,et al.  Development and Validation of the German Work-Related Curiosity Scale , 2012 .

[33]  Leif D. Nelson,et al.  A 21 Word Solution , 2012 .

[34]  R. Peng.  Reproducible Research in Computational Science , 2011, Science.

[35]  G. Cumming.  Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis , 2011 .

[36]  R. Baayen,et al.  Analyzing Reaction Times , 2010 .

[37]  Frank Domahs,et al.  Embodied numerosity: Implicit hand-based representations influence symbolic number processing across cultures , 2010, Cognition.

[38]  Gijsbert Bijlstra,et al.  The social face of emotion recognition: Evaluations versus stereotypes , 2010 .

[39]  Iveta Simera,et al.  EQUATOR: reporting guidelines for health research , 2008, Open medicine : a peer-reviewed, independent, open-access journal.

[40]  M. Pain,et al.  Sprint starts and the minimum auditory reaction time , 2007, Journal of sports sciences.

[41]  W. Dunlap,et al.  Meta-Analysis of Experiments With Matched Groups or Repeated Measures Designs , 1996 .

[42]  P. Jolicoeur,et al.  A Solution to the Effect of Sample Size on Outlier Elimination , 1994 .

[43]  R. Ulrich,et al.  Effects of truncation on reaction time analysis. , 1994, Journal of experimental psychology. General.

[44]  R. Ratcliff.  Methods for dealing with reaction time outliers. , 1993, Psychological bulletin.

[45]  J R Simon,et al.  Processing symbolic information from a visual display: interference from an irrelevant directional cue. , 1970, Journal of experimental psychology.

[46]  R. Viertl.  On the Future of Data Analysis , 2002 .

[47]  R. Duncan Luce,et al.  Response Times: Their Role in Inferring Elementary Mental Organization , 1986 .