The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension

Constituent structure has long been established as a central feature of human language. Analogous to how syntax organizes words in sentences, a narrative grammar organizes sequential images into hierarchic constituents. Here we show that the brain draws upon this constituent structure to comprehend wordless visual narratives. We recorded neural responses as participants viewed sequences of visual images (comics strips) in which blank images either disrupted individual narrative constituents or fell at natural constituent boundaries. A disruption of either the first or the second narrative constituent produced a left-lateralized anterior negativity effect between 500 and 700ms. Disruption of the second constituent also elicited a posteriorly-distributed positivity (P600) effect. These neural responses are similar to those associated with structural violations in language and music. These findings provide evidence that comprehenders use a narrative structure to comprehend visual sequences and that the brain engages similar neurocognitive mechanisms to build structure across multiple domains.

[1]  Gina R. Kuperberg,et al.  Neural mechanisms of language comprehension: Challenges to syntax , 2007, Brain Research.

[2]  M. Garrett,et al.  Syntactically Based Sentence Processing Classes: Evidence from Event-Related Brain Potentials , 1991, Journal of Cognitive Neuroscience.

[3]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[4]  Aniruddh D. Patel,et al.  Language, music, syntax and the brain , 2003, Nature Neuroscience.

[5]  A. Borst.  Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[6]  J. Zacks.  Neuroimaging Studies of Mental Rotation: A Meta-analysis and Review , 2008, Journal of Cognitive Neuroscience.

[7]  Mante S. Nieuwland,et al.  Establishing reference in language comprehension: An electrophysiological perspective , 2007, Brain Research.

[8]  A. Friederici,et al.  Syntactic Gender and Semantic Expectancy: ERPs Reveal Early Autonomy and Late Interaction , 2000, Journal of Cognitive Neuroscience.

[9]  J. Fodor,et al.  The psychological reality of linguistic segments , 1965 .

[10]  M. Rugg,et al.  Event-related potentials and the semantic matching of pictures , 1990, Brain and Cognition.

[11]  Jeffrey M. Zacks,et al.  Prediction Error Associated with the Perceptual Segmentation of Naturalistic Events , 2011, Journal of Cognitive Neuroscience.

[12]  I. Biederman.  Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[13]  Phillip J. Holcomb,et al.  Neurocognitive Mechanisms of Human Comprehension , 2008 .

[14]  Ellen F. Lau,et al.  The role of structural prediction in rapid syntactic analysis , 2006, Brain and Language.

[15]  Sachiko Koyama,et al.  Event-Related Brain Potentials during Memory Search Task , 1993 .

[16]  Jodie A. Baird,et al.  Making sense of human behavior: Action parsing and intentional inference , 2001 .

[17]  Neil Cohn,et al.  Explaining ‘I Can’t Draw’: Parallels between the Structure and Development of Language and Drawing , 2012, Human Development.

[18]  Jeffrey M. Zacks,et al.  Segmentation in reading and film comprehension. , 2009, Journal of experimental psychology. General.

[19]  Peter Ford Dominey,et al.  ERP analysis of cognitive sequencing: a left anterior negativity related to structural transformation processing , 2000, Neuroreport.

[20]  Jeffrey M. Zacks,et al.  Segmentation in the perception and memory of events , 2008, Trends in Cognitive Sciences.

[21]  Peter Hagoort,et al.  How the brain solves the binding problem for language: a neurocomputational model of syntactic processing , 2003, NeuroImage.

[22]  Colin M. Brown,et al.  The syntactic positive shift (sps) as an erp measure of syntactic processing , 1993 .

[23]  Kara D. Federmeier,et al.  So that's what you meant! Event-related potentials reveal multiple aspects of context use during construction of message-level meaning , 2012, NeuroImage.

[24]  William Thompson Price,et al.  The Technique of the Drama , 2008 .

[25]  David I. Perrett,et al.  Event-related potentials and the matching of familiar and unfamiliar faces , 1988, Neuropsychologia.

[26]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[27]  Martin Paczynski,et al.  When Events Change Their Nature: The Neurocognitive Mechanisms Underlying Aspectual Coercion , 2014, Journal of Cognitive Neuroscience.

[28]  P. Holcomb,et al.  Event-related potentials during discourse-level semantic integration of complex pictures. , 2002, Brain research. Cognitive brain research.

[29]  Neil Cohn,et al.  The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. , 2013 .

[30]  Charles A. Perfetti,et al.  An on-line method in studying music parsing , 1993, Cognition.

[31]  W. Beeman.  The Lopsided Ape: Evolution of the Generative Mind , 1994 .

[32]  A. Friederici.  Towards a neural basis of auditory sentence processing , 2002, Trends in Cognitive Sciences.

[33]  Noam Chomsky,et al.  वाक्यविन्यास का सैद्धान्तिक पक्ष = Aspects of the theory of syntax , 1965 .

[34]  Phillip J. Holcomb,et al.  Two Neurocognitive Mechanisms of Semantic Integration during the Comprehension of Visual Real-world Events , 2008, Journal of Cognitive Neuroscience.

[35]  L. Deouell,et al.  ERP evidence for context congruity effects during simultaneous object–scene processing , 2010, Neuropsychologia.

[36]  Thomas A. Farmer,et al.  Early Occipital Sensitivity to Syntactic Category Is Based on Form Typicality , 2010, Psychological science.

[37]  Colin M. Brown,et al.  Syntax-related ERP-effects in Dutch. , 2003, Brain research. Cognitive brain research.

[38]  P. Holcomb,et al.  Event-Related Brain Potentials Reflect Semantic Priming in an Object Decision Task , 1994, Brain and Cognition.

[39]  Morton Ann Gernsbacher,et al.  Surface information loss in comprehension , 1985, Cognitive Psychology.

[40]  Giosuè Baggio,et al.  Computing and recomputing discourse models : An ERP study , 2008 .

[41]  Neil Cohn,et al.  Visual Narrative Structure , 2013, Cogn. Sci..

[42]  D. Rumelhart.  NOTES ON A SCHEMA FOR STORIES , 1975 .

[43]  Aniruddh D. Patel,et al.  Processing Syntactic Relations in Language and Music: An Event-Related Potential Study , 1998, Journal of Cognitive Neuroscience.

[44]  Ovid J. L. Tzeng,et al.  Dynamic allocation of attention to metrical and grouping accents in rhythmic sequences , 2011, Experimental Brain Research.

[45]  Ray Jackendoff,et al.  What is the human language faculty?: Two views , 2011 .

[46]  Thomas F. Mnte,et al.  Brain Activity Associated with Syntactic Incongruencies in Words and Pseudo-Words , 1997, Journal of Cognitive Neuroscience.

[47]  T G Bever,et al.  Segmentation in cinema perception. , 1976, Science.

[48]  E. Maris,et al.  Prior Expectation Mediates Neural Adaptation to Repeated Sounds in the Auditory Cortex: An MEG Study , 2011, The Journal of Neuroscience.

[49]  P. Holcomb,et al.  Event-related brain potentials elicited by syntactic anomaly , 1992 .

[50]  A. Friederici,et al.  Electrophysiological Evidence for Two Steps in Syntactic Analysis: Early Automatic and Late Controlled Processes , 1999, Journal of Cognitive Neuroscience.

[51]  A. Clark.  Whatever next? Predictive brains, situated agents, and the future of cognitive science. , 2013, The Behavioral and brain sciences.

[52]  Neil Cohn,et al.  (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension , 2012, Cognitive Psychology.

[53]  M. Corballis.  The Lopsided Ape: Evolution of the Generative Mind // Review , 1993 .

[54]  W. Ritter,et al.  The scalp topography of potentials associated with missing visual or auditory stimuli. , 1976, Electroencephalography and clinical neurophysiology.

[55]  Roger C. Schank,et al.  SCRIPTS, PLANS, GOALS, AND UNDERSTANDING , 1988 .

[56]  J. Wolfe,et al.  Differential Electrophysiological Signatures of Semantic and Syntactic Scene Processing , 2013, Psychological science.

[57]  N. S. Johnson,et al.  Remembrance of things parsed: Story structure and recall , 1977, Cognitive Psychology.

[58]  A. Kehler,et al.  Verb aspect, event structure, and coreferential processing. , 2009, Journal of memory and language.

[59]  R. Jackendoff.  Language, Consciousness, Culture: Essays on Mental Structure , 2007 .

[60]  Jeffrey M. Zacks,et al.  Human brain activity time-locked to perceptual event boundaries , 2001, Nature Neuroscience.

[61]  Jeffrey M. Zacks,et al.  Perceiving, remembering, and communicating structure in events. , 2001, Journal of experimental psychology. General.

[62]  Phillip J Holcomb,et al.  Semantic integration in videos of real-world events: an electrophysiological investigation. , 2003, Psychophysiology.

[63]  W. Labov,et al.  Narrative analysis: Oral versions of personal experience. , 1997 .

[64]  V. M. Holmes,et al.  Planning units and syntax in sentence production , 1978, Cognition.

[65]  Martin Paczynski,et al.  The difference between "giving a rose" and "giving a kiss": Sustained neural activity to the light verb construction. , 2014, Journal of memory and language.

[66]  I. Winkler,et al.  I Heard That Coming: Event-Related Potential Evidence for Stimulus-Driven Prediction in the Auditory System , 2009, The Journal of Neuroscience.

[67]  Neil Cohn,et al.  You're a Good Structure, Charlie Brown: The Distribution of Narrative Categories in Comic Strips , 2014, Cogn. Sci..

[68]  Stefan Koelsch,et al.  Interaction between Syntax Processing in Language and in Music: An ERP Study , 2005, Journal of Cognitive Neuroscience.

[69]  Aristotle,et al.  The Poetics of Aristotle , 1988 .

[70]  David Kunzle.  History of the comic strip , 1973 .

[71]  E. Donchin,et al.  Is the P300 component a manifestation of context updating? , 1988, Behavioral and Brain Sciences.

[72]  Karl J. Friston,et al.  A theory of cortical responses , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[73]  A. Friederici,et al.  Event-related brain potentials during natural speech processing: effects of semantic, morphological and syntactic violations. , 1993, Brain research. Cognitive brain research.

[74]  付伶俐.  打磨Using Language,倡导新理念 , 2014 .

[75]  Joseph P. Magliano,et al.  The Impact of Continuity Editing in Narrative Film on Event Segmentation , 2011, Cogn. Sci..

[76]  P Finkenzeller,et al.  Evoked responses as a function of external and stored information. , 1968, Electroencephalography and clinical neurophysiology.

[77]  S. Dehaene,et al.  Evidence for a hierarchy of predictions and prediction errors in human cortex , 2011, Proceedings of the National Academy of Sciences.

[78]  Peter Hagoort,et al.  A neurocomputational model of syntactic processing , 2003 .

[79]  Walter A. Siebel,et al.  Opinion TRENDS in Cognitive Sciences Vol.9 No.12 December 2005 Towards , 2022 .

[80]  T G Bever,et al.  Syntactic Structure Modifies Attention during Speech Perception and Recognition , 1969, The Quarterly journal of experimental psychology.

[81]  Mante S. Nieuwland,et al.  Testing the limits of the semantic illusion phenomenon: ERPs reveal temporary semantic change deafness in discourse comprehension. , 2005, Brain research. Cognitive brain research.

[82]  Neil Cohn,et al.  Prediction, events, and the advantage of Agents: The processing of semantic roles in visual narrative , 2013, Cognitive Psychology.

[83]  M. Kutas,et al.  Expect the Unexpected: Event-related Brain Response to Morphosyntactic Violations , 1998 .

[84]  Scott McCloud.  Understanding comics: the invisible art = Memahami komik / Scott McCloud; penerjemah S. Kinanti , 2001 .

[85]  Z S Bond,et al.  Phonological Units in Sentence Perception , 1972, Phonetica.

[86]  Morten H. Christiansen,et al.  Similar neural correlates for language and sequential learning: Evidence from event-related brain potentials , 2012, Language and cognitive processes.

[87]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[88]  R. Jackendoff,et al.  A Generative Theory of Tonal Music , 1985 .