Exploratory and Confirmatory Analyses in Sentence Processing: A Case Study of Number Interference in German.

Given the replication crisis in cognitive science, it is important to consider what researchers need to do in order to report results that are reliable. We consider three changes in current practice that have the potential to deliver more realistic and robust claims. First, the planned experiment should be divided into two stages, an exploratory stage and a confirmatory stage. This clear separation allows the researcher to check whether any results found in the exploratory stage are robust. The second change is to carry out adequately powered studies. We show that this is imperative if we want to obtain realistic estimates of effects in psycholinguistics. The third change is to use Bayesian data-analytic methods rather than frequentist ones; the Bayesian framework allows us to focus on the best estimates we can obtain of the effect, rather than rejecting a strawman null. As a case study, we investigate number interference effects in German. Number feature interference is predicted by cue-based retrieval models of sentence processing (Van Dyke & Lewis, 2003; Vasishth & Lewis, 2006), but it has shown inconsistent results. We show that by implementing the three changes mentioned, suggestive evidence emerges that is consistent with the predicted number interference effects.

[1]  J. V. Van Dyke.  Interference effects from grammatically unavailable constituents during sentence processing. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[2]  E. Jaynes,et al.  Confidence Intervals vs Bayesian Intervals , 1976 .

[3]  J. Carlin,et al.  Beyond Power Calculations , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[4]  Francis Tuerlinckx,et al.  A hierarchical approach for fitting curves to response time measurements , 2008, Psychonomic bulletin & review.

[5]  Andrew Gelman,et al.  The illusion of power: How the statistical significance filter leads to overconfident expectations of replicability , 2017 .

[6]  Shravan Vasishth,et al.  Models of retrieval in sentence comprehension: A computational evaluation using Bayesian hierarchical modeling , 2018, ArXiv.

[7]  Shravan Vasishth,et al.  Statistical Methods for Linguistic Research: Foundational Ideas - Part I , 2016, Lang. Linguistics Compass.

[8]  J. Woolley,et al.  Paradigms and processes in reading comprehension. , 1982, Journal of experimental psychology. General.

[9]  Clinton L. Johns,et al.  Low working memory capacity is only spuriously related to poor reading comprehension , 2014, Cognition.

[10]  Richard L. Lewis,et al.  Computational principles of working memory in sentence comprehension , 2006, Trends in Cognitive Sciences.

[11]  Shravan Vasishth,et al.  What eye movements can tell us about sentence comprehension. , 2013, Wiley interdisciplinary reviews. Cognitive science.

[12]  D. Bates,et al.  Balancing Type I Error and Power in Linear Mixed Models , 2015, 1511.01864.

[13]  D. Barr,et al.  Random effects structure for confirmatory hypothesis testing: Keep it maximal. , 2013, Journal of memory and language.

[14]  S. Vasishth,et al.  Processing Chinese Relative Clauses: Evidence for the Subject-Relative Advantage , 2013, PloS one.

[15]  Roger P. G. van Gompel,et al.  Does number interference occur during sentence processing? , 2012, CogSci.

[16]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[17]  N J Pearlmutter,et al.  Linear versus Hierarchical Agreement Feature Processing in Comprehension , 2000, Journal of psycholinguistic research.

[18]  A. D. de Groot,et al.  The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas] , 2014 .

[19]  G. Logan.  Shapes of reaction-time distributions and shapes of learning curves: a test of the instance theory of automaticity. , 1992, Journal of experimental psychology. Learning, memory, and cognition.

[20]  Shravan Vasishth,et al.  Similarity-based interference in sentence comprehension: Literature review and Bayesian meta-analysis , 2017 .

[21]  Brian Dillon,et al.  Contrasting intrusion profiles for agreement and anaphora: Experimental and modeling evidence , 2013 .

[22]  Shravan Vasishth,et al.  When High-Capacity Readers Slow Down and Low-Capacity Readers Speed Up: Working Memory and Locality Effects , 2016, Front. Psychol..

[23]  H. Schielzeth,et al.  Conclusions beyond support: overconfident estimates in mixed models , 2008, Behavioral ecology : official journal of the International Society for Behavioral Ecology.

[24]  SUNY Stony,et al.  Underspecification of syntactic ambiguities : Evidence from self-paced reading , 2010 .

[25]  D. Bates,et al.  Parsimonious Mixed Models , 2015, 1506.04967.

[26]  Timo B. Roettger,et al.  Using meta-analysis for evidence synthesis: The case of incomplete neutralization in German , 2018, J. Phonetics.

[27]  Margaret L. Kern,et al.  Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach , 2013, PloS one.

[28]  E. Matthew Husband,et al.  Misinterpretations in agreement and agreement attraction , 2016, Quarterly journal of experimental psychology.

[29]  Ellen F. Lau,et al.  Agreement Attraction in Comprehension: Representations and Processes. , 2009 .

[30]  Gabriella Vigliocco,et al.  Subject-verb agreement errors in French and English: The role of syntactic hierarchy , 2002 .

[31]  Richard L. Lewis,et al.  An Activation-Based Model of Sentence Processing as Skilled Memory Retrieval , 2005, Cogn. Sci..

[32]  Jeffrey N. Rouder,et al.  The fallacy of placing confidence in confidence intervals , 2015, Psychonomic bulletin & review.

[33]  D. Cox,et al.  An Analysis of Transformations , 1964 .

[34]  Sol Lagoa,et al.  Agreement Processes in Spanish Comprehension , 2014 .

[35]  Brian A. Nosek,et al.  An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[36]  Jeffrey N. Rouder,et al.  Are unshifted distributional models appropriate for response time? , 2005 .

[37]  R. Kievit,et al.  The meaning of "significance" for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. 1969. , 2014, Acta psychologica.

[38]  Maryellen C. MacDonald,et al.  Plausibility and grammatical agreement , 2003 .

[39]  Karl G. D. Bailey,et al.  Good-Enough Representations in Language Comprehension , 2002 .

[40]  Ellen F. Lau,et al.  Agreement attraction in Spanish comprehension , 2015 .

[41]  Fangfang Li,et al.  Bayesian data analysis in the phonetic sciences: A tutorial introduction , 2018, J. Phonetics.

[42]  Reinhold Kliegl,et al.  Working memory differences in long-distance dependency resolution , 2015, Front. Psychol..

[43]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[44]  Titus von der Malsburg,et al.  Scanpaths reveal syntactic underspecification and reanalysis strategies , 2012, Language and Cognitive Processes.

[45]  Laurel Brehm,et al.  The time-course of feature interference in agreement comprehension: Multiple mechanisms and asymmetrical attraction. , 2014, Journal of memory and language.

[46]  Douglas Saddy,et al.  Processing Negative Polarity Items: When Negation Comes Through the Backdoor , 2005 .

[47]  Jeffrey N. Rouder,et al.  Robust misinterpretation of confidence intervals , 2013, Psychonomic bulletin & review.

[48]  Shravan Vasishth,et al.  Bayesian linear mixed models using Stan: A tutorial for psychologists, linguists, and cognitive scientists , 2015, 1506.06201.

[49]  Reinhold Kliegl,et al.  A Framework for Modeling the Interaction of Syntactic Processing and Eye Movement Control , 2013, Top. Cogn. Sci..

[50]  Shravan Vasishth,et al.  Bayesian Hierarchical Finite Mixture Models of Reading Times: A Case Study , 2017 .

[51]  B. McElree,et al.  Retrieval interference in sentence comprehension. , 2006, Journal of memory and language.

[52]  Susan M. Garnsey,et al.  Agreement Processes in Sentence Comprehension , 1999 .

[53]  Richard L. Lewis,et al.  Argument-Head Distance and Processing Complexity: Explaining both Locality and Antilocality Effects , 2006 .

[54]  B McElree,et al.  Sentence Comprehension Is Mediated by Content-Addressable Memory Structures , 2000, Journal of psycholinguistic research.

[55]  Richard L. Lewis,et al.  Distinguishing effects of structure and decay on attachment and repair: A cue-based parsing account of recovery from misanalyzed ambiguities , 2003 .

[56]  Neal J. Pearlmutter,et al.  Hierarchy and scope of planning in subject–verb agreement production , 2011, Cognition.

[57]  B. McElree,et al.  Cue-dependent interference in comprehension. , 2011 .

[58]  Stephani Foraker,et al.  Memory structures that subserve sentence comprehension , 2003 .

[59]  Titus von der Malsburg,et al.  False Positives and Other Statistical Errors in Standard Analyses of Eye Movements in Reading. , 2015, Journal of memory and language.