Editors’ Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science

Drawn sequences of images are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and they pervade across cultures from instruction manuals to comics. They also appear prevalently as stimuli across Cognitive Science, for studies of temporal cognition, event structure, social cognition, discourse, and basic intelligence. Yet, despite this fundamental place in human expression and research on cognition, the study of visual narratives themselves has only recently gained traction in Cognitive Science. This work has suggested that visual narrative comprehension requires cultural exposure across a developmental trajectory and engages with domain-general processing mechanisms shared by visual perception, attention, event cognition, and language, among others. Here, we review the relevance of such research for the broader Cognitive Science community, and make the case for why researchers should join the scholarship of this ubiquitous but understudied aspect of human expression.

[1]  Neil Cohn.  A multimodal parallel architecture: A cognitive framework for multimodal interactions , 2016, Cognition.

[2]  R M Weist,et al.  The interaction of language and thought in children's language acquisition: a crosslinguistic study , 1997, Journal of Child Language.

[3]  Rolf A. Zwaan,et al.  Generating predictive inferences while viewing a movie , 1996 .

[4]  P. Holcomb,et al.  Event-related potentials during discourse-level semantic integration of complex pictures. , 2002, Brain research. Cognitive brain research.

[5]  Neil Cohn,et al.  A Picture is Worth More Words Over Time: Multimodality and Narrative Structure Across Eight Decades of American Superhero Comics , 2017 .

[6]  Janina Wildfeuer,et al.  An open multilevel classification scheme for the visual layout of comics and graphic novels: Motivation and design , 2016, Digit. Scholarsh. Humanit..

[7]  Panayiota Kendeou,et al.  The Inferential Language Comprehension (iLC) Framework: Supporting Children's Comprehension of Visual Narratives , 2019, Top. Cogn. Sci..

[8]  Lester C. Loschky,et al.  The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives , 2019, Top. Cogn. Sci..

[9]  Neil Cohn,et al.  Explaining ‘I Can’t Draw’: Parallels between the Structure and Development of Language and Drawing , 2012, Human Development.

[10]  Takeshi Okada,et al.  Imitation, Inspiration, and Creation: Cognitive Process of Creative Drawing by Copying Others' Artworks , 2017, Cogn. Sci..

[11]  Charles Forceville,et al.  Visual representations of the idealized cognitive model of anger in the Asterix album 'La Zizanie' , 2005 .

[12]  Neil Cohn,et al.  Navigating Comics: An Empirical and Theoretical Approach to Strategies of Reading Comic Page Layouts , 2013, Front. Psychol..

[13]  Lester C. Loschky,et al.  What is the role of the film viewer? The effects of narrative comprehension and viewing task on gaze control in film , 2017, Cognitive research: principles and implications.

[14]  M. Kutas,et al.  The Search for Common Sense: An Electrophysiological Study of the Comprehension of Words and Pictures in Reading , 1996, Journal of Cognitive Neuroscience.

[15]  Neil Cohn,et al.  Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension , 2019, Top. Cogn. Sci..

[16]  Joseph P. Magliano,et al.  Sources of Complexity in Narrative Comprehension across Media , 2019, Narrative Complexity.

[17]  Neil Cohn,et al.  Visual Narrative Structure , 2013, Cogn. Sci..

[18]  Brent Wilson,et al.  An Iconoclastic View of the Imagery Sources in the Drawings of Young People. , 1977 .

[19]  M. Kutas,et al.  When a hit sounds like a kiss: An electrophysiological exploration of semantic processing in visual narrative , 2017, Brain and Language.

[20]  D. Tanner,et al.  A strong wink between verbal and emoji-based irony: How the brain processes ironic emojis during language comprehension , 2018, PloS one.

[21]  C. Caltagirone,et al.  The role played by the right hemisphere in the organization of complex textual structures , 2005, Brain and Language.

[22]  Chantal E. Stern,et al.  Evidence for the importance of basal ganglia output nuclei in semantic event sequencing: An fMRI study , 2006, Brain Research.

[23]  Neil Cohn,et al.  Drawing the Line Between Constituent Structure and Coherence Relations in Visual Narratives , 2017, Journal of experimental psychology. Learning, memory, and cognition.

[24]  Nancy L. Monson,et al.  Picture sequencing by schizophrenic patients , 1993 .

[25]  N. S. Johnson,et al.  Remembrance of things parsed: Story structure and recall , 1977, Cognitive Psychology.

[26]  Neil Cohn,et al.  The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension , 2014, Neuropsychologia.

[27]  Patricia Baggett.  Structurally equivalent stories in movie and text and the effect of the medium on recall. , 1979 .

[28]  J. Hosler,et al.  Are Comic Books an Effective Way to Engage Nonmajors in Learning and Appreciating Science?1 , 2011, CBE - Life Sciences Education.

[29]  J. Bateman,et al.  A multimodal discourse theory of visual narrative , 2014 .

[30]  T. Smith.  The attentional theory of cinematic continuity , 2012 .

[31]  Morten H. Christiansen,et al.  Similar neural correlates for language and sequential learning: Evidence from event-related brain potentials , 2012, Language and cognitive processes.

[32]  T. Sørensen,et al.  Is threat the only modulator of attentional selectivity? Redefining the Easterbrook hypothesis , 2014, Front. Psychol..

[33]  B. Gordon,et al.  Visual and linguistic narrative comprehension in autism spectrum disorders: Neural evidence for modality-independent impairments , 2018, Brain and Language.

[34]  Christine Liddel.  Every picture tells a story: South African and British children interpreting pictures , 1996 .

[35]  Aniruddh D. Patel,et al.  Language, music, syntax and the brain , 2003, Nature Neuroscience.

[36]  Daniel L. Dinsmore,et al.  “Reading” Paintings: Evidence for Trans-Symbolic and Symbol-Specific Comprehension Processes , 2015 .

[37]  Simpson W. L. Wong,et al.  Graphic Novel Comprehension Among Learners with Differential Cognitive Styles and Reading Abilities , 2017 .

[38]  Neil Cohn,et al.  The pieces fit: Constituent structure and global coherence of visual narrative in RSVP. , 2016, Acta psychologica.

[39]  Lester C. Loschky,et al.  Understanding Moment‐to‐Moment Processing of Visual Narratives , 2018, Cogn. Sci..

[40]  Sigal Eden,et al.  Enhancing Sequential Time Perception and Storytelling Ability of Deaf and Hard of Hearing Children , 2011, American annals of the deaf.

[41]  Charles Forceville,et al.  Pictorial runes in Tintin and the Picaros , 2011 .

[42]  P. Boggio,et al.  Listening beyond seeing: Event-related potentials to audiovisual processing in visual narrative , 2018, Brain and Language.

[43]  Tom Trabasso,et al.  The Development of Goal Plans of Action in the Narration of a Picture Story. , 1992 .

[44]  S. Baron-Cohen,et al.  Mechanical, behavioural and Intentional understanding of picture stories in autistic children , 1986 .

[45]  Emily L Coderre,et al.  Dismantling the "Visual Ease Assumption: " A Review of Visual Narrative Processing in Clinical Populations , 2020, Top. Cogn. Sci..

[46]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[47]  Jochen Laubrock,et al.  Computational Approaches to Comics Analysis , 2019, Top. Cogn. Sci..

[48]  Tom Foulsham,et al.  Meaning above the head: combinatorial constraints on the visual vocabulary of comics , 2016 .

[49]  Maurizio Corbetta,et al.  Unravelling nonverbal cognitive performance in acquired aphasia , 2009 .

[50]  Howard Gardner,et al.  Comprehension of humorous and nonhumorous materials by left and right brain-damaged patients , 1986, Brain and Cognition.

[51]  Stephan Schwan,et al.  First-time viewers' comprehension of films: bridging shot transitions. , 2015, British journal of psychology.

[52]  David M. McCord,et al.  The WAIS-R Comprehension and Picture Arrangement Subtests as Measures of Social Intelligence: Testing Traditional Interpretations , 1996 .

[53]  Douglas P. Newton.  Children's Perception of Pictorial Metaphor , 1985 .

[54]  Chris Garforth,et al.  Research and testing non-formal education materials: a multi-media extension project in Botswana. , 1980 .

[55]  B BREIGER.  The use of the W-B picture arrangement subtest as a projective technique. , 1956, Journal of consulting psychology.

[56]  Rebecca M. Todd,et al.  Iconic faces are not real faces: enhanced emotion detection and altered neural processing as faces become more iconic , 2016, Cognitive research: principles and implications.

[57]  Joseph P. Magliano,et al.  Filling in the Gaps: Memory Implications for Inferring Missing Content in Graphic Narratives , 2017 .

[58]  Hitoshi Kiya,et al.  Integrated Model of Image Protection Techniques , 2018, J. Imaging.

[59]  Ariella Lehrer,et al.  The relationship between reading and cognitive processing of television and radio , 1984 .

[60]  Walter Huber,et al.  Linguistic and nonlinguistic processing of narratives in aphasia , 1982, Brain and Language.

[61]  Joseph P. Magliano,et al.  Aging and perceived event structure as a function of modality , 2012, Neuropsychology, development, and cognition. Section B, Aging, neuropsychology and cognition.

[62]  B. L. Cook.  Picture Communication in Papua New Guinea. , 1980 .

[63]  Michał Szawerna.  Metaphoricity of Conventionalized Diegetic Images in Comics , 2017 .

[64]  Joseph P. Magliano,et al.  Chapter 9 Toward a Comprehensive Model of Comprehension , 2009 .

[65]  Matthias Kümmerer,et al.  Attention to Comics , 2018, Empirical Comics Research.

[66]  T. Trabasso,et al.  Logical necessity and transitivity of causal relations in stories , 1989 .

[67]  Ann L. Brown,et al.  Construction and Regeneration of Logical Sequences Using Causes or Consequences as the Point of Departure. , 1976 .

[68]  Morton Ann Gernsbacher,et al.  Surface information loss in comprehension , 1985, Cognitive Psychology.

[69]  Jean-Christophe Burie,et al.  Knowledge-driven understanding of images in comic books , 2015, International Journal on Document Analysis and Recognition (IJDAR).

[70]  Larry R Price,et al.  The joint WAIS-III and WMS-III factor structure: development and cross-validation of a six-factor model of cognitive functioning. , 2003, Psychological assessment.

[71]  W. Friedman,et al.  Children's representations of the pattern of daily activities. , 1990, Child development.

[72]  Neil Cohn,et al.  You're a Good Structure, Charlie Brown: The Distribution of Narrative Categories in Comic Strips , 2014, Cogn. Sci..

[73]  Neil Cohn,et al.  Action starring narratives and events: Structure and inference in visual narrative comprehension , 2015, Journal of cognitive psychology.

[74]  A. Rumsey,et al.  Getting the story straight: Language fieldwork using a narrative problem-solving task , 2012 .

[75]  Phillip Vaughan,et al.  Reading Words and Images , 2018, Empirical Comics Research.

[76]  T. Trabasso,et al.  Constructing inferences during narrative text comprehension. , 1994 .

[77]  Edith Kaan,et al.  Event-Related Potentials and Language Processing: A Brief Overview , 2007, Lang. Linguistics Compass.

[78]  Marta Kutas,et al.  Getting a cue before getting a clue: Event-related potentials to inference in visual narrative comprehension , 2015, Neuropsychologia.

[79]  Hitoshi Isahara,et al.  Story Pattern Analysis Based on Scene Order Information in Four-Scene Comics , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).

[80]  Olivier Le Guen,et al.  No Metaphorical Timeline in Gesture and Cognition Among Yucatec Mayas , 2012, Front. Psychology.

[81]  Miško Subotić,et al.  Cognitive and electrophysiological characteristics of children with specific language impairment and subclinical epileptiform electroencephalogram , 2014, Journal of clinical and experimental neuropsychology.

[82]  W. Beatty,et al.  Picture and motor sequencing in multiple sclerosis. , 1994, Journal of clinical and experimental neuropsychology.

[83]  Neil Cohn,et al.  (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension , 2012, Cognitive Psychology.

[84]  M. Bornens,et al.  Problems brought about by "reading" a sequence of pictures. , 1990, Journal of experimental child psychology.

[85]  Jean-Christophe Burie,et al.  Comic Characters Detection Using Deep Learning , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).

[86]  J MCFIE,et al.  Intellectual impairment with localized cerebral lesions. , 1952, Brain : a journal of neurology.

[87]  Lester C. Loschky,et al.  The relative roles of visuospatial and linguistic working memory systems in generating inferences during visual narrative comprehension , 2015, Memory & Cognition.

[88]  Francisco O. D. Veloso,et al.  On the track of visual style: a diachronic study of page composition in comics and its functional motivation , 2019, Visual Communication.

[89]  Patricia Howlin,et al.  Brief Report: Assessment of the Social-Emotional Profile in Children with Autism Spectrum Disorders using a Novel Comic Strip Task , 2012, Journal of autism and developmental disorders.

[90]  John A Bateman,et al.  From Empirical Studies to Visual Narrative Organization , 2018, Empirical Comics Research.

[91]  S. Kunen,et al.  Figural factors and the development of pictorial inferences. , 1987, Journal of experimental child psychology.

[92]  Hitoshi Isahara,et al.  Estimation of structure of four-scene comics by convolutional neural networks , 2016, MANPU@ICPR.

[93]  Joseph P. Magliano,et al.  Detecting Differences between Adapted Narratives , 2018, Empirical Comics Research.

[94]  Dorothy V. M. Bishop,et al.  The role of syntax in encoding and recall of pictorial narratives: Evidence from specific language impairment , 2005 .

[95]  Paul E. Jose,et al.  Cultural differences in Chinese American and European American children's drawing skills over time , 2011 .

[96]  Neil Cohn,et al.  The changing pages of comics : Page layouts across eight decades of American superhero comics , 2016 .

[97]  A. H. Die,et al.  The Wais-R Picture Arrangement Subtest: What do Scores Indicate? , 1986 .

[98]  Joshua D. Lipsitz,et al.  Wechsler Comprehension and Picture Arrangement Subtests and Social Adjustment. , 1993 .

[99]  Neil Cohn,et al.  The notion of the motion: The neurocognition of motion lines in visual narratives , 2015, Brain Research.

[100]  Eakta Jain,et al.  A preliminary benchmark of four saliency algorithms on comic art , 2016, 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).