Prediction, events, and the advantage of Agents: The processing of semantic roles in visual narrative

Agents consistently appear prior to Patients in sentences, manual signs, and drawings, and Agents are responded to faster when presented in visual depictions of events. We hypothesized that this "Agent advantage" reflects Agents' role in event structure. We investigated this question by manipulating the depictions of Agents and Patients in preparatory actions in wordless visual narratives. We found that Agents elicited a greater degree of predictions regarding upcoming events than Patients, that Agents are viewed longer than Patients, independent of serial order, and that visual depictions of actions are processed more quickly following the presentation of an Agent vs. a Patient. Taken together these findings support the notion that Agents initiate the building of event representation. We suggest that Agent First orders facilitate the interpretation of events as they unfold and that the saliency of Agents within visual representations of events is driven by anticipation of upcoming events.

[1]  Edward L. Keenan,et al.  Towards a universal definition of "Subject , 2014 .

[2]  S. Goldin‐Meadow.  The Resilience of Language : What Gesture Creation in Deaf Children Can Tell Us About How All Children Learn Language , 2005 .

[3]  Eysenck Keane.  Cognitive Psychology, 6 , 2012 .

[4]  G. R. Potts,et al.  Assessing the occurrence of elaborative inferences: Lexical decision versus naming , 1988 .

[5]  Robert Forkel,et al.  The World Atlas of Language Structures Online , 2009 .

[6]  Herbert Schriefers,et al.  Journal of Memory and Language , 2001 .

[7]  John A. Hawkins,et al.  Word order universals , 1983 .

[8]  Joseph P. Magliano,et al.  When goals collide: Monitoring the goals of multiple characters , 2005, Memory & cognition.

[9]  Jeffrey M. Zacks,et al.  Segmentation in the perception and memory of events , 2008, Trends in Cognitive Sciences.

[10]  David A. Zubin.  Discourse Function of Morphology: The Focus System in German , 1979 .

[11]  J. Zacks.  Neuroimaging Studies of Mental Rotation: A Meta-analysis and Review , 2008, Journal of Cognitive Neuroscience.

[12]  Lisa Gershkoff-Stowe,et al.  Is there a natural order for expressing semantic relations? , 2002, Cognitive Psychology.

[13]  M. Glanzer,et al.  Short-term storage in reading. , 1984 .

[14]  David N. Rapp,et al.  Integrating Memory-Based and Constructionist Processes in Accounts of Reading Comprehension , 2005 .

[15]  A. Chatterjee.  Disembodying cognition , 2010, Language and Cognition.

[16]  George A. Miller,et al.  A Chronometric Study of Some Relations between Sentences , 1964 .

[17]  W. Marslen-Wilson,et al.  The temporal structure of spoken language understanding , 1980, Cognition.

[18]  A. Chatterjee,et al.  Verbs, events and spatial representations , 1999, Neuropsychologia.

[19]  Celia M. Klin,et al.  When Throwing a Vase Has Multiple Consequences: Minimal Encoding of Predictive Inferencest , 2003 .

[20]  J G Greeno,et al.  Time to read semantically related sentences* , 1974, Memory & cognition.

[21]  Wolfgang Sternefeld,et al.  Wortstellung und Informationsstruktur: Eine performanzbezogene Erklärung von Topik- und Fokus-Positionen Word Order and Information Structure: A Performance Based Account of Topic Positions and Focus Positions , 1993 .

[22]  S. Goldin-Meadow,et al.  Gestural communication in deaf children: noneffect of parental input on language development. , 1983, Science.

[23]  Joseph H. Greenberg,et al.  Some Universals of Grammar with Particular Reference to the Order of Meaningful Elements , 1990, On Language.

[24]  David E. Kieras,et al.  Good and bad structure in simple paragraphs: Effects on apparent theme, reading time, and recal , 1978 .

[25]  Giuliano Geminiani,et al.  Analogical representation and language structure , 1995, Neuropsychologia.

[26]  A. Michotte,et al.  La perception de la causalite , 1956 .

[27]  M. Silverstein.  7. Hierarchy of Features and Ergativity , 1986 .

[28]  H. Sasse.  The thetic/categorical distinction revisited , 1987 .

[29]  K. Verfaillie,et al.  The priority of the agent in visual event perception: On the cognitive basis of grammatical agent-patient asymmetries , 1996 .

[30]  N. Segalowitz.  The perception of semantic relations in pictures , 1982, Memory & cognition.

[31]  Michael Meeuwis,et al.  Order of subject, object, and verb , 2013 .

[32]  Neil Cohn.  Visual Language Panels Time Temporal Map the Limits of Time and Transitions: Challenges to Theories of Sequential Image Comprehension , 2022 .

[33]  Rebecca Fincher-Kiefer,et al.  The role of predictive inferences in situation model construction , 1993 .

[34]  S Goldin-Meadow,et al.  The development of language-like communication without a language model. , 1977, Science.

[35]  Neil Cohn,et al.  (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension , 2012, Cognitive Psychology.

[36]  M. Kutas,et al.  Influences of semantic and syntactic context on open- and closed-class words , 1991, Memory & cognition.

[37]  M. Gernsbacher,et al.  Investigating differences in general comprehension skill. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[38]  Judith Aissen,et al.  Markedness and Subject Choice in Optimality Theory , 1999 .

[39]  Anjan Chatterjee,et al.  Language and space: some interactions , 2001, Trends in Cognitive Sciences.

[40]  By Neil Cohn,et al.  A Visual Lexicon , 2007 .

[41]  Fritz Heider,et al.  La perception de la causalit , 1947 .

[42]  Herbert H. Clark,et al.  Some structural properties of simple active and passive sentences , 1965 .

[43]  David Kemmerer,et al.  The Cross-Linguistic Prevalence of SOV and SVO Word Orders Reflects the Sequential and Hierarchical Representation of Action in Broca's Area , 2012, Lang. Linguistics Compass.

[44]  Morton Ann Gernsbacher,et al.  Surface information loss in comprehension , 1985, Cognitive Psychology.

[45]  Rolf A. Zwaan,et al.  Generating predictive inferences while viewing a movie , 1996 .

[46]  Jeffrey M. Zacks,et al.  Human brain activity time-locked to perceptual event boundaries , 2001, Nature Neuroscience.

[47]  Morton Ann Gernsbacher,et al.  Language Comprehension As Structure Building , 1990 .

[48]  H. Schriefers,et al.  Prediction in language comprehension beyond specific words: An ERP study on sentence comprehension in Polish , 2013 .

[49]  Susan Goldin.  The resilience of language : what gesture creation in deaf children can tell us about how all children learn language , 2003 .

[50]  Neil Cohn,et al.  Visual Narrative Structure , 2013, Cogn. Sci..

[51]  Jeffrey M. Zacks,et al.  Event perception: a mind-brain perspective. , 2007, Psychological bulletin.

[52]  M. Gernsbacher,et al.  Accessing Sentence Participants: The Advantage of First Mention. , 1988, Journal of memory and language.

[53]  Robin K. Morris,et al.  Processing Subject and Object Relative Clauses: Evidence from Eye Movements , 2002 .

[54]  Rebecca Fincher-Kiefer,et al.  The Role of Visuospatial Resources in Generating Predictive and Bridging Inferences , 2004 .

[55]  Ray Jackendoff,et al.  Semantic Interpretation in Generative Grammar , 1972 .

[56]  Fernanda Ferreira,et al.  The misinterpretation of noncanonical sentences , 2003, Cognitive Psychology.

[57]  Anne Maass,et al.  Directional Bias in the Mental Representation of Spatial Events , 2003, Psychological science.

[58]  Arthur C. Graesser,et al.  The time course of generating causal antecedent and causal consequence inferences , 1993 .

[59]  J. Trueswell,et al.  Getting the gist of events: recognition of two-participant actions from brief displays. , 2013, Journal of experimental psychology. General.

[60]  Phillip J. Holcomb,et al.  Two Neurocognitive Mechanisms of Semantic Integration during the Comprehension of Visual Real-world Events , 2008, Journal of Cognitive Neuroscience.

[61]  G. Suci,et al.  Event perception by children in the early stages of language production. , 1980, Child development.

[62]  Jeffrey M. Zacks,et al.  Prediction Error Associated with the Perceptual Segmentation of Naturalistic Events , 2011, Journal of Cognitive Neuroscience.

[63]  Christian Dobel,et al.  How Writing System and Age Influence Spatial Representations of Actions , 2007, Psychological science.

[64]  Rebecca Fincher-Kiefer,et al.  Encoding differences between bridging and predictive inferences , 1996 .

[65]  R. Ratcliff,et al.  Inferences about predictable events. , 1986, Journal of experimental psychology. Learning, memory, and cognition.

[66]  Filip Germeys,et al.  The psychology of film: perceiving beyond the cut , 2007, Psychological research.

[67]  J. Mandler,et al.  On the psychological validity of story structure , 1982 .

[68]  S. Goldin-Meadow,et al.  The natural order of events: How speakers of different languages represent events nonverbally , 2008, Proceedings of the National Academy of Sciences.

[69]  Margot D. Lasher.  The cognitive representation of an event involving human motion , 1981, Cognitive Psychology.

[70]  P. Tannenbaum,et al.  Generation of active and passive sentences as a function of subject or object focus , 1968 .