Comics on the Brain: Structure and Meaning in Sequential Image Comprehension

Comics on the Brain: Structure and Meaning in Sequential Image Comprehension Neil Cohn (neil.cohn@tufts.edu), Martin Paczynski, Phil Holcomb Department of Psychology, 490 Boston Ave Medford, MA 02155 USA Ray Jackendoff Center for Cognitive Studies, Miner Hall Medford, MA 02155 USA Gina Kuperberg Department of Psychiatry and Athinoula A. Martinos Center for Biomedical Imaging Massachusetts General Hospital, Bldg 149, 13th Street Charlestown, MA 02129 USA Abstract Background Research on language comprehension has long drawn a distinction between meaning and structure. Most theoretical work pertaining to this separation has been at the level of the single sentence (Chomsky, 1965). However, this general divide is also relevant our understanding of how we make sense of text and discourse, beyond the single sentence. Much work on text and discourse has emphasized the local relationships which draw together related concepts into common semantic fields (Halliday & Hasan, 1985; van Dijk, 1977) or through the establishment of coherence relationships across individual sentences (Halliday & Hasan, 1976; Zwaan & Radvansky, 1998). Theories of narrative with a more global scope have also been articulated in “story grammars” (e.g. Mandler & Johnson, 1977; Rumelhart, 1975; Thorndyke, 1977), which base individual narrative categories around characters’ goals. These categories were organized within a formalistic generative grammar which imposed a global structure for understanding how characters navigated through events. Similar to verbal discourse, most theories describing sequential image structure have also focused on linear semantic relationships between individual images (McCloud, 1993). Recently though, Cohn (2003, In prep) has proposed a theoretical model for the structure of graphic narratives. Like story grammars’ treatment of Just as syntax differentiates coherent sentences from scrambled word strings, sequential images must also use a cognitive system to distinguish coherent narratives from random strings of images. We conducted experiments analogous to two classic psycholinguistic studies to examine structure and semantics in sequential images. We compared Normal comic strips with both structure and meaning to sequences with Semantics Only, Structure Only, or randomly Scrambled panels. Experiment 1 used a target-monitoring paradigm, and found that RTs were slowest to target panels in Scrambled sequences, intermediate in Structural Only and Semantic Only sequences, and fastest in Normal sequences. Experiment 2 measured ERPs to the same strips. The largest N400 appeared in Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in Normal sequences. Together, these findings suggest that sequential image comprehension is guided by an interaction between a structure and meaning, broadly analogous to syntax and semantics in language. Keywords: Narrative; Comics; Visual Language; Discourse; ERPs; N400 Introduction Drawings have been conveying narratives through sequences of images for millennia, whether painted on cave walls, carved into reliefs, hung on medieval tapestries, or, in their modern context, appearing in comic books (McCloud, 1993). Compared to research on the structure and comprehension of verbal narrative, however, few studies have examined the driving forces behind understanding this visual language: what are the representations and mechanisms engaged during sequential image comprehension? And how is the meaning integrated with structure across a sequence of images?