False Positives and Other Statistical Errors in Standard Analyses of Eye Movements in Reading.

In research on eye movements in reading, it is common to analyze a number of canonical dependent measures to study how the effects of a manipulation unfold over time. Although this gives rise to the well-known multiple comparisons problem, i.e. an inflated probability that the null hypothesis is incorrectly rejected (Type I error), it is accepted standard practice not to apply any correction procedures. Instead, there appears to be a widespread belief that corrections are not necessary because the increase in false positives is too small to matter. To our knowledge, no formal argument has ever been presented to justify this assumption. Here, we report a computational investigation of this issue using Monte Carlo simulations. Our results show that, contrary to conventional wisdom, false positives are increased to unacceptable levels when no corrections are applied. Our simulations also show that counter-measures like the Bonferroni correction keep false positives in check while reducing statistical power only moderately. Hence, there is little reason why such corrections should not be made a standard requirement. Further, we discuss three statistical illusions that can arise when statistical power is low, and we show how power can be improved to prevent these illusions. In sum, our work renders a detailed picture of the various types of statistical errors than can occur in studies of reading behavior and we provide concrete guidance about how these errors can be avoided.

[1]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[2]  E. Wagenmakers A practical solution to the pervasive problems ofp values , 2007, Psychonomic bulletin & review.

[3]  Keith Rayner,et al.  Parafoveal-foveal overlap can facilitate ongoing word identification during reading: evidence from eye movements. , 2013, Journal of experimental psychology. Human perception and performance.

[4]  Shravan Vasishth,et al.  What eye movements can tell us about sentence comprehension. , 2013, Wiley interdisciplinary reviews. Cognitive science.

[5]  Reinhold Kliegl,et al.  SWIFT: a dynamical model of saccade generation during reading. , 2005, Psychological review.

[6]  R. Kliegl,et al.  Adult age differences in the perceptual span during reading. , 2011, Psychology and aging.

[7]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[8]  Reinhold Kliegl,et al.  A Framework for Modeling the Interaction of Syntactic Processing and Eye Movement Control , 2013, Top. Cogn. Sci..

[9]  A. Gelman,et al.  The garden of forking paths : Why multiple comparisons can be a problem , even when there is no “ fishing expedition ” or “ p-hacking ” and the research hypothesis was posited ahead of time ∗ , 2019 .

[10]  S. Holm A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[11]  Brian A. Nosek,et al.  Recommendations for Increasing Replicability in Psychology † , 2013 .

[12]  Erik D. Reichle,et al.  Toward a model of eye movement control in reading. , 1998, Psychological review.

[13]  J. Kruschke What to believe: Bayesian methods for data analysis , 2010, Trends in Cognitive Sciences.

[14]  Shravan Vasishth,et al.  Statistical Methods for Linguistic Research: Foundational Ideas - Part I , 2016, Lang. Linguistics Compass.

[15]  D. Barr,et al.  Random effects structure for confirmatory hypothesis testing: Keep it maximal. , 2013, Journal of memory and language.

[16]  K. Rayner,et al.  Eye movements in reading words and sentences , 2007 .

[17]  K. Rayner Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[18]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[19]  Paul E. Smaldino,et al.  Replication, Communication, and the Population Dynamics of Scientific Discovery , 2015, PloS one.

[20]  K. Rayner,et al.  Toward a model of eye movement control in reading. , 1998 .

[21]  J. Carlin,et al.  Beyond Power Calculations , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[22]  Shravan Vasishth,et al.  Statistical Methods for Linguistic Research: Foundational Ideas - Part I , 2016, Lang. Linguistics Compass.

[23]  Shravan Vasishth,et al.  The Importance of Reading Naturally: Evidence From Combined Recordings of Eye Movements and Electric Brain Potentials. , 2017, Cognitive science.

[24]  Andy B. Yoo,et al.  Approved for Public Release; Further Dissemination Unlimited X-ray Pulse Compression Using Strained Crystals X-ray Pulse Compression Using Strained Crystals , 2002 .