Action starring narratives and events: Structure and inference in visual narrative comprehension

Studies of discourse have long placed focus on the inference generated by information that is not overtly expressed, and theories of visual narrative comprehension similarly focused on the inference generated between juxtaposed panels. Within the visual language of comics, star-shaped “flashes” commonly signify impacts, but can be enlarged to the size of a whole panel that can omit all other representational information. These “action star” panels depict a narrative culmination (a “Peak”), but have content which readers must infer, thereby posing a challenge to theories of inference generation in visual narratives that focus only on the semantic changes between juxtaposed images. This paper shows that action stars demand more inference than depicted events, and that they are more coherent in narrative sequences than scrambled sequences (Experiment 1). In addition, action stars play a felicitous narrative role in the sequence (Experiment 2). Together, these results suggest that visual narratives use conventionalized depictions that demand the generation of inferences while retaining narrative coherence of a visual sequence.

[1]  W. Kintsch,et al.  Strategies of discourse comprehension , 1983 .

[2]  R. Brooke Lea,et al.  Have We Been Searching for Meaning in All the Wrong Places? Defining the "Search After Meaning" Principle in Comprehension , 2005 .

[3]  Rolf A. Zwaan,et al.  Chapter 18 – Discourse Comprehension , 2006 .

[4]  J. Mandler Language comprehension. , 1979, Science.

[5]  D. Barr,et al.  Random effects structure for confirmatory hypothesis testing: Keep it maximal. , 2013, Journal of memory and language.

[6]  R. Ratcliff,et al.  Inference during reading. , 1992, Psychological review.

[7]  Francisco Yus Ramos Inferring from comics: a multi-stage account , 2008 .

[8]  Edward Branigan Narrative Comprehension and Film , 1992 .

[9]  R. Ratcliff,et al.  Inferences about predictable events. , 1986, Journal of experimental psychology. Learning, memory, and cognition.

[10]  Arthur C. Graesser,et al.  Dimensions of situation model construction in narrative comprehension. , 1995 .

[11]  S. Garrod,et al.  Understanding written language: Explorations of comprehension beyond the sentence , 1981 .

[12]  M. Rugg,et al.  Event-related potentials and the semantic matching of pictures , 1990, Brain and Cognition.

[13]  D. Rumelhart NOTES ON A SCHEMA FOR STORIES , 1975 .

[14]  Neil Cohn,et al.  You're a Good Structure, Charlie Brown: The Distribution of Narrative Categories in Comic Strips , 2014, Cogn. Sci..

[15]  Robert F. Williams,et al.  Image Schemas and Conceptual Metaphor in Action Comics , 2012 .

[16]  W. Kintsch The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[17]  Mandler JeanM. A code in the node: The use of a story schema in retrieval , 1978 .

[18]  Richard Taylor,et al.  The Poetics of cinema , 1982 .

[19]  Mario Saraceni Language beyond language: comics as verbo-visual texts , 2001 .

[20]  K. Rayner,et al.  The Effect of Clause Wrap-Up on Eye Movements during Reading , 2000, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[21]  Neil Cohn,et al.  (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension , 2012, Cognitive Psychology.

[22]  T. Trabasso,et al.  Causal relatedness and importance of story events , 1985 .

[23]  David Bordwell Narration in the Fiction Film , 1985 .

[24]  Jeffrey M. Zacks,et al.  Event structure in perception and conception. , 2001, Psychological bulletin.

[25]  N. S. Johnson,et al.  Remembrance of things parsed: Story structure and recall , 1977, Cognitive Psychology.

[26]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[27]  Michael Halliday,et al.  Cohesion in English , 1976 .

[28]  Joseph P. Magliano,et al.  The Impact of Continuity Editing in Narrative Film on Event Segmentation , 2011, Cogn. Sci..

[29]  S. Eisenstein,et al.  The Film Sense , 1942 .

[30]  G. Radvansky,et al.  Functionality and spatial relations in memory and language , 2000, Memory & cognition.

[31]  William C. Mann,et al.  RHETORICAL STRUCTURE THEORY: A THEORY OF TEXT ORGANIZATION , 1987 .

[32]  Neil Cohn,et al.  The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension , 2014, Neuropsychologia.

[33]  David I. Perrett,et al.  Event-related potentials and the matching of familiar and unfamiliar faces , 1988, Neuropsychologia.

[34]  Arthur C. Graesser,et al.  The time course of generating causal antecedent and causal consequence inferences , 1993 .

[35]  P. Broek Comprehension and memory of narrative texts: Inferences and coherence. , 1994 .

[36]  P. Holcomb,et al.  Event-related potentials during discourse-level semantic integration of complex pictures. , 2002, Brain research. Cognitive brain research.

[37]  Michael S. Wogalter,et al.  Comprehension and Memory , 1999 .

[38]  Neil Cohn,et al.  The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. , 2013 .

[39]  M. Singer,et al.  Validation of causal bridging inferences in discourse understanding. , 1992 .

[40]  David N. Rapp,et al.  Focusing effects from online and offline reading tasks. , 2011 .

[41]  Vincent M. Reid,et al.  N400 involvement in the processing of action sequences , 2008, Neuroscience Letters.

[42]  Rolf A. Zwaan,et al.  Generating predictive inferences while viewing a movie , 1996 .

[43]  Jerome L. Myers,et al.  Accessing the discourse representation during reading , 1998 .

[44]  J. Keenan,et al.  The effects of causal cohesion on comprehension and memory , 1984 .

[45]  Joseph P. Magliano,et al.  Chapter 9 Toward a Comprehensive Model of Comprehension , 2009 .

[46]  Abel G. Oliva,et al.  Gist of a scene , 2005 .

[47]  J. Hobbs On the coherence and structure of discourse , 1985 .

[48]  D. Bates,et al.  Linear Mixed-Effects Models using 'Eigen' and S4 , 2015 .

[49]  Phillip J. Holcomb,et al.  Two Neurocognitive Mechanisms of Semantic Integration during the Comprehension of Visual Real-world Events , 2008, Journal of Cognitive Neuroscience.

[50]  S. Duffy,et al.  Degree of causal relatedness and memory , 1987 .

[51]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[52]  H. H. Clark,et al.  What's new? Acquiring New information as a process in comprehension , 1974 .

[53]  M. Glanzer,et al.  Short-term storage in reading. , 1984 .

[54]  J. Mandler,et al.  Is There More than One Way to Recall a Story , 1979 .

[55]  Neil Cohn,et al.  The notion of the motion: The neurocognition of motion lines in visual narratives , 2015, Brain Research.

[56]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[57]  S. Chatman,et al.  Story and Discourse , 2019 .

[58]  B. A. Conway,et al.  The effects of laforin, malin, Stbd1, and Ptg deficiencies on heart glycogen levels in Pompe disease mouse models , 2015 .

[59]  N. Stein,et al.  An Analysis of Story Comprehension in Elementary School Children: A Test of a Schema. , 1975 .

[60]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[61]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[62]  T. Trabasso Causal Cohesion and Story Coherence. , 1982 .

[63]  Morton Ann Gernsbacher,et al.  Language Comprehension As Structure Building , 1990 .

[64]  Mario Saraceni,et al.  Relatedness: Aspects of Textual Connectivity in Comics , 2016, The Visual Narrative Reader.

[65]  Rolf A. Zwaan,et al.  Indexing space and time in film understanding , 2001 .

[66]  Neil Cohn,et al.  Visual Narrative Structure , 2013, Cogn. Sci..

[67]  Jeffrey M. Zacks,et al.  Segmentation in reading and film comprehension. , 2009, Journal of experimental psychology. General.

[68]  Charles Sorbie The Notion of Motion , 2005 .

[69]  Rolf A. Zwaan,et al.  Discourse comprehension. , 1997, Annual review of psychology.

[70]  Neil Cohn Structure, Meaning, and Constituency in Visual Narrative Comprehension , 2012 .

[71]  J. Mandler Stories, Scripts, and Scenes: Aspects of Schema Theory , 1984 .

[72]  P. Thorndyke Cognitive structures in comprehension and memory of narrative discourse , 1977, Cognitive Psychology.

[73]  Neil Cohn,et al.  Prediction, events, and the advantage of Agents: The processing of semantic roles in visual narrative , 2013, Cognitive Psychology.

[74]  Andrew Kehler,et al.  Coherence, reference, and the theory of grammar , 2002, CSLI lecture notes series.

[75]  A. D. Manning,et al.  Understanding Comics: The Invisible Art , 1993 .

[76]  H. Mack Coherence And Structure , 1974 .

[77]  Nancy L. Stein,et al.  The effects of organization and instructional set on story memory , 1978 .

[78]  M. Gernsbacher,et al.  Investigating differences in general comprehension skill. , 1990, Journal of experimental psychology. Learning, memory, and cognition.