Multimodal Event Knowledge in Online Sentence Comprehension: the Influence of Visual Context on Anticipatory Eye Movements

People predict incoming words during online sentence comprehension based on their knowledge of real-world events that is cued by preceding linguistic contexts. We used the visual world paradigm to investigate how event knowledge activated by an agent-verb pair is integrated with perceptual information about the referent that fits the patient role. During the verb time window participants looked significantly more at the referents that are expected given the agent-verb pair. Results are consistent with the assumption that event-based knowledge involves perceptual properties of typical participants. The knowledge activated by the agent is compositionally integrated with knowledge cued by the verb to drive anticipatory eye movements during sentence comprehension based on the expectations associated not only with the incoming word, but also with the visual features of its referent.

[1]  L. Barsalou.  Grounded cognition. , 2008, Annual review of psychology.

[2]  Yuki Kamide,et al.  Anticipatory Processes in Sentence Processing , 2008, Lang. Linguistics Compass.

[3]  Katrin Erk,et al.  A Structured Vector Space Model for Word Meaning in Context , 2008, EMNLP.

[4]  D. Barr.  Random effects structure for testing interactions in linear mixed-effects models , 2013, Front. Psychol..

[5]  G. Altmann,et al.  Incremental interpretation at verbs: restricting the domain of subsequent reference , 1999, Cognition.

[6]  Jelena Mirkovic,et al.  Incrementality and Prediction in Human Sentence Processing , 2009, Cogn. Sci..

[7]  Emmanuele Chersoni,et al.  Measuring Thematic Fit with Distributional Feature Overlap , 2017, EMNLP.

[8]  G. Altmann,et al.  The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing , 2007 .

[9]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[10]  J. Elman.  On the Meaning of Words and Dinosaur Bones: Lexical Knowledge Without a Lexicon , 2009, Cogn. Sci..

[11]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[12]  Falk Huettig,et al.  The tug of war between phonological, semantic and shape information in language-mediated visual search , 2007 .

[13]  J. Elman,et al.  Effects of event knowledge in processing verbal arguments. , 2010, Journal of memory and language.

[14]  Gina R Kuperberg,et al.  What do we mean by prediction in language comprehension? , 2016, Language, cognition and neuroscience.

[15]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[16]  M. Tanenhaus,et al.  Modeling the Influence of Thematic Fit (and Other Constraints) in On-line Sentence Comprehension , 1998 .

[17]  Pia Knoeferle,et al.  Visually Situated Language Comprehension , 2016, Lang. Linguistics Compass.

[18]  Gerry T. M. Altmann,et al.  Thematic role assignment in context , 1999 .

[19]  Yuki Kamide,et al.  Now you see it, now you don't: mediating the mapping between language and the visual world , 2004 .

[20]  Christoph Scheepers,et al.  Event-based plausibility immediately influences on-line language comprehension. , 2011, Journal of experimental psychology. Learning, memory, and cognition.

[21]  Alessandro Lenci,et al.  Distributional Models of Word Meaning , 2018 .

[22]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[23]  Elia Bruni,et al.  Multimodal Distributional Semantics , 2014, J. Artif. Intell. Res..

[24]  Alessandro Lenci,et al.  Composing and Updating Verb Argument Expectations: A Distributional Semantic Model , 2011, CMCL@ACL.

[25]  Douwe Kiela.  MMFeat: A Toolkit for Extracting Multi-Modal Features , 2016, ACL.

[26]  Angeliki Lazaridou,et al.  Combining Language and Vision with a Multimodal Skip-gram Model , 2015, NAACL.

[27]  J. Elman,et al.  Generalized event knowledge activation during online sentence comprehension. , 2012, Journal of memory and language.

[28]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[29]  Matthew W. Crocker,et al.  The Coordinated Interplay of Scene, Utterance, and World Knowledge: Evidence From Eye Tracking , 2006, Cogn. Sci..

[30]  Katrin Erk,et al.  A Flexible, Corpus-Driven Model of Regular and Inverse Selectional Preferences , 2010, CL.

[31]  G. Altmann,et al.  The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements , 2003 .

[32]  J. Elman,et al.  A basis for generating expectancies for verbs from nouns , 2005, Memory & cognition.

[33]  Matthew W. Crocker,et al.  The influence of recent scene events on spoken comprehension: Evidence from eye movements , 2007 .

[34]  Mary Hare,et al.  Activating event knowledge , 2009, Cognition.

[35]  Matthew W. Crocker,et al.  The influence of the immediate visual context on incremental thematic role-assignment: evidence from eye-movements in depicted events , 2005, Cognition.

[36]  K. McRae,et al.  Integrating Verbs, Situation Schemas, and Thematic Role Concepts , 2001 .