A Picture is Worth More Words Over Time: Multimodality and Narrative Structure Across Eight Decades of American Superhero Comics

Abstract The visual narratives of comics involve complex multimodal interactions between written language and the visual language of images, where one or the other may guide the meaning and/or narrative structure. We investigated this interaction in a corpus analysis across eight decades of American superhero comics (1940–2010s). No change across publication date was found for multimodal interactions that weighted meaning towards text or across both text and images, where narrative structures were present across images. However, we found an increase over time of narrative sequences with meaning weighted to the visuals, and an increase of sequences without text at all. These changes coincided with an overall reduction in the number of words per panel, a shift towards panel framing with single characters and close-ups rather than whole scenes, and an increase in shifts between temporal states between panels. These findings suggest that storytelling has shifted towards investing more information in the images, along with an increasing complexity and maturity of the visual narrative structures. This has shifted American comics from being textual stories with illustrations to being visual narratives that use text.

[1]  Edith Kaan,et al.  Event-Related Potentials and Language Processing: A Brief Overview , 2007, Lang. Linguistics Compass.

[2]  Neil Cohn,et al.  The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. , 2013 .

[3]  Matthew A. Cicci.  Turning The Page: Fandoms, Multimodality, And The Transformation Of The 'comic Book' Superhero , 2015 .

[4]  Neil Cohn,et al.  Narrative conjunction's junction function: The interface of narrative grammar and semantics in sequential images , 2015 .

[5]  J. Cutting,et al.  Attention and the Evolution of Hollywood Film , 2010, Psychological science.

[6]  Dusan Stamenkovic,et al.  The Interplay of Words and Images in Expressing Multimodal Metaphors in Comics , 2015 .

[7]  Alexander Danner,et al.  Comics: A Global History, 1968 to the Present , 2014 .

[8]  Neil Cohn,et al.  The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension , 2014, Neuropsychologia.

[9]  Andrew Salway,et al.  A system for image–text relations in new (and old) media , 2005 .

[10]  Tony Veale,et al.  Balloonics: the visuals of balloons in comics , 2010 .

[11]  John A. Bateman,et al.  Text and Image , 2014 .

[12]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[13]  Neil Cohn,et al.  Pow, Punch, Pika, and Chu: The Structure of Sound Effects in Genres of American Comics and Japanese Manga , 2016 .

[14]  M. Gernsbacher,et al.  Investigating differences in general comprehension skill. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[15]  M. Osaka,et al.  Serial changes of humor comprehension for four-frame comic Manga: an fMRI study , 2014, Scientific Reports.

[16]  Neil Cohn,et al.  The vocabulary of manga: Visual morphology in dialects of Japanese Visual Language , 2016 .

[17]  Joseph P. Magliano,et al.  The Impact of Continuity Editing in Narrative Film on Event Segmentation , 2011, Cogn. Sci..

[18]  A. D. Manning,et al.  Understanding Comics: The Invisible Art , 1993 .

[19]  Neil Cohn,et al.  Framing Attention in Japanese and American Comics: Cross-Cultural Differences in Attentional Structure , 2012, Front. Psychology.

[20]  Neil Cohn,et al.  A different kind of cultural frame: An analysis of panels in American comics and Japanese manga , 2011 .

[21]  Stephan Schwan,et al.  Watching Film for the First Time , 2010, Psychological science.

[22]  Rolf A. Zwaan,et al.  Indexing space and time in film understanding , 2001 .

[23]  Terry Royce.  Intersemiotic complementarity: A framework for multimodal discourse analysis , 2013 .

[24]  Neil Cohn,et al.  Visual Narrative Structure , 2013, Cogn. Sci..

[25]  Jeannett Martin,et al.  Reading Visual Narratives: Image Analysis of Children's Picture Books , 2012 .

[26]  Neil Cohn,et al.  The changing pages of comics : Page layouts across eight decades of American superhero comics , 2016 .

[27]  Lester C. Loschky,et al.  The relative roles of visuospatial and linguistic working memory systems in generating inferences during visual narrative comprehension , 2015, Memory & Cognition.

[28]  R. Petersen.  Comics, manga, and graphic novels : a history of graphic narratives , 2011 .

[29]  K. Worcester,et al.  The Superhero Reader , 2013 .

[30]  Neil Cohn,et al.  Drawing the Line Between Constituent Structure and Coherence Relations in Visual Narratives , 2017, Journal of experimental psychology. Learning, memory, and cognition.

[31]  S Goldin-Meadow,et al.  Silence is liberating: removing the handcuffs on grammatical expression in the manual modality. , 1996, Psychological review.

[32]  Jennifer Green,et al.  Drawn from the Ground: Sound, Sign and Inscription in Central Australian Sand Stories , 2014 .

[33]  Charles Forceville,et al.  Pictorial runes in Tintin and the Picaros , 2011 .

[34]  Neil Cohn.  A multimodal parallel architecture: A cognitive framework for multimodal interactions , 2016, Cognition.

[35]  T. Foulsham,et al.  Reading Without Words: Eye Movements in the Comprehension of Comic Strips , 2016 .

[36]  Janina Wildfeuer,et al.  Defining units of analysis for the systematic analysis of comics: A discourse-based approach , 2014 .

[37]  S. Goldin‐Meadow.  Talking and Thinking With Our Hands , 2006 .

[38]  Neil Cohn,et al.  (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension , 2012, Cognitive Psychology.

[39]  J. Cutting.  The Framing of Characters in Popular Movies , 2015 .

[40]  J. Bateman,et al.  A multimodal discourse theory of visual narrative , 2014 .

[41]  Sean A. Guynes.  Four-Color Sound: A Peircean Semiotics of Comic Book Onomatopoeia , 2014 .

[42]  Marta Kutas,et al.  Getting a cue before getting a clue: Event-related potentials to inference in visual narrative comprehension , 2015, Neuropsychologia.

[43]  Francis Lacassin.  The Comic Strip and Film Language , 1972 .

[44]  Charles Forceville,et al.  Visual representation of emotion in manga: Loss of control is Loss of hands in Azumanga Daioh Volume 4 , 2011 .

[45]  Neil Cohn,et al.  The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics , 2014, Front. Psychol..