Reading Without Words: Eye Movements in the Comprehension of Comic Strips

Copyright © 2016 John Wiley & Sons, Ltd. The study of attention in pictures is mostly limited to individual images. When we ‘read’ a visual narrative (e.g., a comic strip), the pictures have a coherent sequence, but it is not known how this affects attention. In two experiments, we eyetracked participants in order to investigate how disrupting the visual sequence of a comic strip would affect attention. Both when panels were presented one at a time (Experiment 1) and when a sequence was presented all together (Experiment 2), pictures were understood more quickly and with fewer fixations when in their original order. When order was randomised, the same pictures required more attention and additional ‘regressions’. Fixation distributions also differed when the narrative was intact, showing that context affects where we look. This reveals the role of top-down structures when we attend to pictorial information, as well as providing a springboard for applied research into attention within image sequences.Copyright © 2016 John Wiley & Sons, Ltd.

[1]  John Morton,et al.  The Effects of Context upon Speed of Reading, Eye Movements and Eye-voice Span , 1964 .

[2]  J. Morton.  THE EFFECTS OF CONTEXT ON THE VISUAL DURATION THRESHOLD FOR WORDS. , 1964, British journal of psychology.

[3]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[4]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[5]  M. Glanzer,et al.  Short-term storage in reading. , 1984 .

[6]  Morton Ann Gernsbacher,et al.  Surface information loss in comprehension , 1985, Cognitive Psychology.

[7]  M. Gernsbacher,et al.  Investigating differences in general comprehension skill. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[8]  Morton Ann Gernsbacher,et al.  Language Comprehension As Structure Building , 1990 .

[9]  P. Carroll,et al.  Visual Analysis of Cartoons: A View from the Far Side , 1992 .

[10]  M. Just,et al.  Constructing mental models of machines from text and diagrams. , 1993 .

[11]  J. Henderson,et al.  The effects of semantic consistency on eye movements during complex scene viewing , 1999 .

[12]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[13]  K. Rayner,et al.  The Effect of Clause Wrap-Up on Eye Movements during Reading , 2000, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[14]  Andrew J. Stewart,et al.  Integrating text and pictorial information: eye movements when looking at print advertisements. , 2001, Journal of experimental psychology. Applied.

[15]  Scott McCloud.  Understanding comics: the invisible art = Memahami komik / Scott McCloud; penerjemah S. Kinanti , 2001 .

[16]  G. Cumming,et al.  A Primer on the Understanding, Use, and Calculation of Confidence Intervals that are Based on Central and Noncentral Distributions , 2001 .

[17]  David S Wooding,et al.  Eye movements of large populations: II. Deriving regions of interest, coverage, and similarity using fixation maps , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[18]  Richard P. DeShon,et al.  Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. , 2002, Psychological methods.

[19]  Derrick J. Parkhurst,et al.  Modeling the role of salience in the allocation of overt visual attention , 2002, Vision Research.

[20]  A. Kennedy,et al.  Parafoveal-on-foveal effects in normal reading , 2005, Vision Research.

[21]  Takahide Omori・Takeharu Igaki.  Eye catchers in comics: Controlling eye movements in reading pictorial and textual media , 2005 .

[22]  T. Foulsham,et al.  Quarterly Journal of Experimental Psychology: in press Visual saliency and semantic incongruency influence eye movements when , 2022 .

[23]  Antonio Torralba,et al.  Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. , 2006, Psychological review.

[24]  Ronald A. Rensink.  Scene Perception , 2007 .

[25]  D. Pelli,et al.  Parts, Wholes, and Context in Reading: A Triple Dissociation , 2007, PloS one.

[26]  Thierry Groensteen,et al.  The System of Comics , 2007 .

[27]  K. Holmqvist,et al.  Reading Information Graphics: the Role of Spatial Contiguity and Dual Attentional Guidance , 2022 .

[28]  George L. Malcolm,et al.  Searching in the dark: Cognitive relevance drives attention in real-world scenes , 2009, Psychonomic bulletin & review.

[29]  K. Rayner.  The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search , 2009, Quarterly journal of experimental psychology.

[30]  John M. Henderson,et al.  Clustering of Gaze During Dynamic Scene Viewing is Predicted by Motion , 2011, Cognitive Computation.

[31]  Tom Foulsham,et al.  Saccade control in natural images is shaped by the information visible at fixation: evidence from asymmetric gaze-contingent windows , 2011, Attention, perception & psychophysics.

[32]  Michael D. Dodd,et al.  Examining the influence of task set on eye movements and fixations. , 2011, Journal of vision.

[33]  Tom Foulsham,et al.  Scrambled eyes? Disrupting scene structure impedes focal processing and increases bottom-up guidance , 2011, Attention, perception & psychophysics.

[34]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[35]  Daniel T. Levin,et al.  A Window on Reality , 2012 .

[36]  Neil Cohn,et al.  (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension , 2012, Cognitive Psychology.

[37]  T. Smith.  The attentional theory of cinematic continuity , 2012 .

[38]  Neil Cohn.  Comics, Linguistics, and Visual Language: The Past and Future of a Field , 2012 .

[39]  Neil Cohn,et al.  Prediction, events, and the advantage of Agents: The processing of semantic roles in visual narrative , 2013, Cognitive Psychology.

[40]  Neil Cohn,et al.  Visual Narrative Structure , 2013, Cogn. Sci..

[41]  Neil Cohn,et al.  The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. , 2013 .

[42]  Kai Kunze,et al.  I know what you are reading: recognition of document types using mobile eye tracking , 2013, ISWC '13.

[43]  Barbara Postema.  Narrative Structure in Comics: Making Sense of Fragments , 2013 .

[44]  Kenneth C. Scott-Brown,et al.  Behavioral biases when viewing multiplexed scenes: scene structure and frames of reference for inspection , 2013, Front. Psychol..

[45]  Neil Cohn,et al.  Navigating Comics: An Empirical and Theoretical Approach to Strategies of Reading Comic Page Layouts , 2013, Front. Psychol..

[46]  Neil Cohn,et al.  The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics , 2014, Front. Psychol..

[47]  D. Rosenthal,et al.  Quality-space theory in olfaction , 2014, Front. Psychol..

[48]  Neil Cohn,et al.  The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension , 2014, Neuropsychologia.

[49]  Neil Cohn,et al.  You're a Good Structure, Charlie Brown: The Distribution of Narrative Categories in Comic Strips , 2014, Cogn. Sci..

[50]  J. Bateman,et al.  A multimodal discourse theory of visual narrative , 2014 .

[51]  Neil Cohn,et al.  Action starring narratives and events: Structure and inference in visual narrative comprehension , 2015, Journal of cognitive psychology.

[52]  Neil Cohn,et al.  Navigating Comics II: Constraints on the Reading Order of Comic Page Layouts , 2015 .

[53]  Behavioral Biases , 2018, Behavioral Finance for Private Banking.

[54]  A Components of Sentence and IU Word Reading Times , 2018 .