Filling in the Gaps: Memory Implications for Inferring Missing Content in Graphic Narratives

Visual narratives, including graphic novels, illustrated instructions, and picture books, convey event sequences constituting a plot but cannot depict all events that make up the plot. Viewers must generate inferences that fill the gaps between explicitly shown images. This study explored the inferential products and memory implications of processing gaps in visual narratives. Participants viewed picture-stories containing event sequences comprised of beginning, bridging, and end states and the presence of these event state panels was systematically manipulated. The pattern of processing times after omitted event panels suggests that participants inferred missing beginning and bridging states but not missing end states. In a recognition test including both seen and unseen event panels, participants' memories were most accurate for end state events. The results suggest that generating inferences distorts memory for explicit content, particularly content that has a high semantic overlap with potentially constructed inferences.

[1]  Rolf A. Zwaan.  The Immersed Experiencer: Toward An Embodied Theory Of Language Comprehension , 2003 .

[2]  Rolf A. Zwaan,et al.  Generating predictive inferences while viewing a movie , 1996 .

[3]  T. Trabasso,et al.  Constructing inferences during narrative text comprehension. , 1994, Psychological review.

[4]  H Stanislaw,et al.  Calculation of signal detection theory measures , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[5]  Benjamin Naumann,et al.  Mental Representations A Dual Coding Approach , 2016 .

[6]  David S. Miall.  Experiencing narrative worlds: On the psychological activities of reading , 2000 .

[7]  A. Graesser,et al.  The Generation of Knowledge - Based Inferences During Narrative Comprehension , 1985 .

[8]  Lawrence W. Barsalou,et al.  Perceptions of perceptual symbols , 1999, Behavioral and Brain Sciences.

[9]  Jessica Daecher,et al.  Frog Where Are You , 2016 .

[10]  M. Gernsbacher,et al.  Investigating differences in general comprehension skill. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[11]  Neil Cohn,et al.  Action starring narratives and events: Structure and inference in visual narrative comprehension , 2015, Journal of cognitive psychology.

[12]  Joseph P. Magliano,et al.  Assessing Reading Skill With a Think-Aloud Procedure and Latent Semantic Analysis , 2003 .

[13]  Morton Ann Gernsbacher,et al.  Language Comprehension As Structure Building , 1990 .

[14]  J. Schooler,et al.  Verbal overshadowing of visual memories: Some things are better left unsaid , 1990, Cognitive Psychology.

[15]  Mercer Mayer,et al.  A Boy, a Dog, and a Frog , 1979 .

[16]  S. Kosslyn,et al.  Imagery, propositions, and the form of internal representations , 1977, Cognitive Psychology.

[17]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[18]  Arthur C. Graesser,et al.  The impact of a schema on comprehension and memory , 1982 .

[19]  Neil Cohn,et al.  (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension , 2012, Cognitive Psychology.

[20]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[21]  Murray Singer,et al.  Constructing and Validating Motive Bridging Inferences , 1996, Cognitive Psychology.

[22]  T. Trabasso,et al.  Inferences during Reading: Converging Evidence from Discourse Analysis, Talk-Aloud Protocols, and Recognition Priming , 1993 .

[23]  Arthur C. Graesser,et al.  The time course of generating causal antecedent and causal consequence inferences , 1993 .

[24]  F. Schmalhofer,et al.  Three components of understanding a programmer's manual: Verbatim, propositional and situational representations , 1986 .

[25]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[26]  Stephan Schwan,et al.  Watching Film for the First Time , 2010, Psychological science.

[27]  Joseph P. Magliano.  Beyond Language Comprehension : Situation Models as a Form of Autobiographical Memory , 2006 .

[28]  T. Trabasso,et al.  Modeling causal integration and availability of information during comprehension of narrative texts. , 1999 .

[29]  A. D. Manning,et al.  Understanding Comics: The Invisible Art , 1993 .

[30]  W. Kintsch.  The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[31]  David N. Rapp,et al.  Focusing effects from online and offline reading tasks. , 2011 .

[32]  Mercer Mayer,et al.  A Boy, a Dog, a Frog, and a Friend , 1967 .

[33]  Neil Cohn,et al.  The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension , 2014, Neuropsychologia.

[34]  Lester C. Loschky,et al.  The relative roles of visuospatial and linguistic working memory systems in generating inferences during visual narrative comprehension , 2015, Memory & Cognition.

[35]  J. G. Snodgrass,et al.  Pragmatics of measuring recognition memory: applications to dementia and amnesia. , 1988, Journal of experimental psychology. General.

[36]  G J Hitch,et al.  Influence of short-term memory codes on visual image processing: evidence from image transformation tasks. , 1992, Journal of experimental psychology. Learning, memory, and cognition.

[37]  M. Louwerse.  Embodied relations are encoded in language , 2008, Psychonomic bulletin & review.

[38]  Rolf A. Zwaan,et al.  Language Encodes Geographical Information , 2009, Cogn. Sci..

[39]  A Craig,et al.  Nonparametric Measures of Sensory Efficiency for Sustained Monitoring Tasks , 1979, Human factors.

[40]  D. Bates,et al.  Mixed-Effects Models in S and S-PLUS , 2001 .

[41]  Joseph P. Magliano,et al.  When goals collide: Monitoring the goals of multiple characters , 2005, Memory & cognition.

[42]  Rolf A. Zwaan,et al.  Indexing space and time in film understanding , 2001 .

[43]  Rolf A. Zwaan.  Embodiment and language comprehension: reframing the discussion , 2014, Trends in Cognitive Sciences.

[44]  David N. Rapp,et al.  Noticing and Revising Discrepancies as Texts Unfold , 2009 .

[45]  Jeffrey Heer,et al.  Narrative Visualization: Telling Stories with Data , 2010, IEEE Transactions on Visualization and Computer Graphics.

[46]  David Bordwell.  Narration in the Fiction Film , 1985 .

[47]  Mercer Mayer,et al.  Frog on His Own , 1973 .

[48]  Patricia Baggett.  Structurally equivalent stories in movie and text and the effect of the medium on recall. , 1979 .

[49]  Robert Leeson,et al.  One frog too many , 1991 .

[50]  Joseph P. Magliano,et al.  Chapter 9 Toward a Comprehensive Model of Comprehension , 2009 .

[51]  Panayiota Kendeou,et al.  Revising what readers know: Updating text representations during narrative comprehension , 2007, Memory & cognition.

[52]  D. LaBerge,et al.  Basic processes in reading : perception and comprehension , 2017 .

[53]  T. Trabasso,et al.  Causal thinking and the representation of narrative events , 1985 .

[54]  Neil Cohn,et al.  You're a Good Structure, Charlie Brown: The Distribution of Narrative Categories in Comic Strips , 2014, Cogn. Sci..

[55]  Charles R. Fletcher,et al.  Surface forms, textbases, and situation models: Recognition memory for three types of textual information , 1990 .

[56]  John D. Bransford,et al.  The abstraction of linguistic ideas , 1971 .