The influence of recent scene events on spoken comprehension: Evidence from eye movements

Evidence from recent experiments that monitored attention in clipart scenes during spoken comprehension suggests that people preferably rely on non-stereotypical depicted events over stereotypical thematic knowledge for incremental interpretation. The Coordinated Interplay Account [Knoeferle, P., & Crocker, M. W. (2006). The coordinated interplay of scene, utterance, and world knowledge: evidence from eye tracking. Cognitive Science, 30, 481–529.] accounts for this preference through referential processing (e.g., the verb mediates a depicted event) and the preferred use of scene event information that is associated with the referent (e.g., the agent of the depicted event). Three eye-tracking experiments examined the generality of this account. While the rapid use of depicted events was replicated in all three studies, the preference to rely on them was modulated by the decay of events that were no longer co-present. Our findings motivate the extension of the Coordinated Interplay Account with an explicit working memory. The coordinated interplay mechanism together with working memory and decay, is shown to account for the influence of scene-derived versus stored knowledge both when events are co-present and when they have recently been perceived.

[1]  Marshall R. Mayberry,et al.  A Connectionist Model of the Coordinated Interplay of Scene, Utterance, and World Knowledge , 2006 .

[2]  D. C. Howell Statistical Methods for Psychology , 1987 .

[3]  Fernanda Ferreira,et al.  Scene Perception for Psycholinguists. , 2004 .

[4]  E. Marshall,et al.  NIMH: caught in the line of fire without a general , 1995, Science.

[5]  M. Tanenhaus,et al.  Actions and affordances in syntactic ambiguity resolution. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[6]  Daniel C. Richardson,et al.  Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension , 2005, Cogn. Sci..

[7]  J. Trueswell,et al.  The role of discourse context in the processing of a flexible word-order language , 2004, Cognition.

[8]  Julia Grant,et al.  Relations between the non-verbal context of maternal speech and rate of language development , 1986 .

[9]  Daniel C. Richardson,et al.  Thinking outside the brain: Spatial indices to visual and linguistic Information. , 2004 .

[10]  Matthew W. Crocker,et al.  The Coordinated Interplay of Scene, Utterance, and World Knowledge: Evidence From Eye Tracking , 2006, Cogn. Sci..

[11]  Michael J. Spivey,et al.  Oculomotor mechanisms activated by imagery and memory: eye movements to absent objects , 2001, Psychological research.

[12]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[13]  C. A. Ferguson,et al.  Talking to Children: Language Input and Acquisition , 1979 .

[14]  Daniel C. Richardson,et al.  Representation, space and Hollywood Squares: looking at things that aren't there anymore , 2000, Cognition.

[15]  Pia Knoeferle,et al.  Comparing the time-course of processing initially ambiguous and unambiguous German SVO/OVS sentences in depicted events , 2007 .

[16]  G. Altmann Language-mediated eye movements in the absence of a visual world: the ‘blank screen paradigm’ , 2004, Cognition.

[17]  G. Altmann,et al.  Incremental interpretation at verbs: restricting the domain of subsequent reference , 1999, Cognition.

[18]  Philip J. Dunham,et al.  Joint-Attentional States and Lexical Acquisition at 18 Months. , 1993 .

[19]  Christoph Scheepers,et al.  Integration of Syntactic and Semantic Information in Predictive Processing: Cross-Linguistic Evidence from German and English , 2003, Journal of psycholinguistic research.

[20]  Rajesh P. N. Rao,et al.  Embodiment is the foundation, not a level , 1996, Behavioral and Brain Sciences.

[21]  M. D’Esposito Working memory. , 2008, Handbook of clinical neurology.

[22]  R. Jackendoff Foundations of Language: Brain, Meaning, Grammar, Evolution , 2002 .

[23]  Matthew W. Crocker,et al.  The influence of the immediate visual context on incremental thematic role-assignment: evidence from eye-movements in depicted events , 2005, Cognition.

[24]  R. H. Baayen,et al.  The CELEX Lexical Database (CD-ROM) , 1996 .

[25]  Daniel C. Richardson,et al.  The Art of Conversation Is Coordination , 2007, Psychological science.

[26]  Zenzi M. Griffin,et al.  Why Look? Reasons for Eye Movements Related to Language Production. , 2004 .

[27]  Julie C. Sedivy,et al.  Achieving incremental semantic interpretation through contextual representation , 1999, Cognition.