What is the role of the film viewer? The effects of narrative comprehension and viewing task on gaze control in film

Film is ubiquitous, but the processes that guide viewers’ attention while viewing film narratives are poorly understood. In fact, many film theorists and practitioners disagree on whether the film stimulus (bottom-up) or the viewer (top-down) is more important in determining how we watch movies. Reading research has shown a strong connection between eye movements and comprehension, and scene perception studies have shown strong effects of viewing tasks on eye movements, but such idiosyncratic top-down control of gaze in film would be anathema to the universal control mainstream filmmakers typically aim for. Thus, in two experiments we tested whether the eye movements and comprehension relationship similarly held in a classic film example, the famous opening scene of Orson Welles’ Touch of Evil (Welles & Zugsmith, Touch of Evil, 1958). Comprehension differences were compared with more volitionally controlled task-based effects on eye movements. To investigate the effects of comprehension on eye movements during film viewing, we manipulated viewers’ comprehension by starting participants at different points in a film, and then tracked their eyes. Overall, the manipulation created large differences in comprehension, but only produced modest differences in eye movements. To amplify top-down effects on eye movements, a task manipulation was designed to prioritize peripheral scene features: a map task. This task manipulation created large differences in eye movements when compared to participants freely viewing the clip for comprehension. Thus, to allow for strong, volitional top-down control of eye movements in film, task manipulations need to make features that are important to narrative comprehension irrelevant to the viewing task. The evidence provided by this experimental case study suggests that filmmakers’ belief in their ability to create systematic gaze behavior across viewers is confirmed, but that this does not indicate universally similar comprehension of the film narrative.

[1]  Tim J. Smith,et al.  Watching you watch movies: Using eye tracking to inform cognitive film theory. , 2013 .

[2]  Surya Saha,et al.  Survey of Endosymbionts in the Diaphorina citri Metagenome and Assembly of a Wolbachia wDi Draft Genome , 2012, PloS one.

[3]  E. Wagenmakers,et al.  A default Bayesian hypothesis test for correlations and partial correlations , 2012, Psychonomic bulletin & review.

[4]  Graham L. Pierce,et al.  Eye movements during scene viewing: Evidence for mixed control of fixation durations , 2008, Psychonomic bulletin & review.

[5]  Lester C. Loschky,et al.  The relative roles of visuospatial and linguistic working memory systems in generating inferences during visual narrative comprehension , 2015, Memory & Cognition.

[6]  G. Radvansky.  Situation models, propositions, and the fan effect , 2005, Psychonomic bulletin & review.

[7]  K Rayner,et al.  Elaborative inferences during reading: do they occur on-line? , 1988, Journal of experimental psychology. Learning, memory, and cognition.

[8]  George L. Malcolm,et al.  Searching in the dark: Cognitive relevance drives attention in real-world scenes , 2009, Psychonomic bulletin & review.

[9]  Christopher J. Honey,et al.  Same Story , Different Story : The Neural Representation of Interpretive Frameworks , 2017 .

[10]  Uri Hasson,et al.  Temporal eye movement strategies during naturalistic viewing. , 2012, Journal of vision.

[11]  Rolf A. Zwaan,et al.  Generating predictive inferences while viewing a movie , 1996 .

[12]  Jerome L. Myers,et al.  Accessing the discourse representation during reading , 1998 .

[13]  Ronan G. Reilly,et al.  Some empirical tests of an interactive activation model of eye movement control in reading , 2006, Cognitive Systems Research.

[14]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[15]  Eyal M. Reingold,et al.  Saccadic Inhibition in Voluntary and Reflexive Saccades , 2002, Journal of Cognitive Neuroscience.

[16]  Andrew Hollingworth,et al.  Eye Movements During Scene Viewing: An Overview , 1998 .

[17]  K. Rayner,et al.  Toward a model of eye movement control in reading. , 1998 .

[18]  R. Walker,et al.  A model of saccade generation based on parallel processing and competitive inhibition , 1999, Behavioral and Brain Sciences.

[19]  Mikko Sams,et al.  Synchronous brain activity across individuals underlies shared psychological perspectives , 2014, NeuroImage.

[20]  Asha Iyer,et al.  Components of bottom-up gaze allocation in natural images , 2005, Vision Research.

[21]  P. Webb.  Making movies. , 1987, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[22]  R Ratcliff,et al.  Memory-based language processing: psycholinguistic research in the 1990s. , 1998, Annual review of psychology.

[23]  Sandra Lowe,et al.  Film Technique And Film Acting , 2016 .

[24]  David Bordwell,et al.  Post-Theory: Reconstructing Film Studies , 1997 .

[25]  A. Kingstone,et al.  Gaze selection in complex social scenes , 2008 .

[26]  Asif A. Ghazanfar,et al.  Human-Monkey Gaze Correlations Reveal Convergent and Divergent Patterns of Movie Viewing , 2010, Current Biology.

[27]  Sébastien Miellet,et al.  iMap: a novel method for statistical fixation mapping of eye movement data , 2011, Behavior research methods.

[28]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[29]  David Windridge,et al.  Looking to Score: The Dissociation of Goal Influence on Eye Movement and Meta-Attentional Allocation in a Complex Dynamic Natural Scene , 2012, PloS one.

[30]  Joseph P. Magliano,et al.  Chapter 9 Toward a Comprehensive Model of Comprehension , 2009 .

[31]  J. Helmert,et al.  Visual Fixation Durations and Saccade Amplitudes: Shifting Relationship in a Variety of Conditions , 2008 .

[32]  R. Buckner,et al.  Evidence for the Default Network's Role in Spontaneous Cognition , 2010 .

[33]  Peter J. Walters.  Mind, brain and narrative , 2017 .

[34]  Marcus Nyström,et al.  Smooth pursuit detection in binocular eye-tracking data with automatic video-based performance evaluation. , 2016, Journal of vision.

[35]  M A Just,et al.  A theory of reading: from eye fixations to comprehension. , 1980, Psychological review.

[36]  Joseph P. Magliano,et al.  The Impact of Continuity Editing in Narrative Film on Event Segmentation , 2011, Cogn. Sci..

[37]  Thomas Martinetz,et al.  Variability of eye movements when viewing dynamic natural scenes. , 2010, Journal of vision.

[38]  D. Heeger,et al.  Reliability of cortical activity during natural stimulation , 2010, Trends in Cognitive Sciences.

[39]  Eileen Kowler,et al.  Anticipatory smooth eye movements with random-dot kinematograms. , 2012, Journal of vision.

[40]  David Bordwell,et al.  Intensified Continuity Visual Style in Contemporary American Film , 2002 .

[41]  R. Malach,et al.  Intersubject Synchronization of Cortical Activity During Natural Vision , 2004, Science.

[42]  S. Eisenstein,et al.  The Film Sense , 1942 .

[43]  T. Foulsham,et al.  How Does the Purpose of Inspection Influence the Potency of Visual Salience in Scene Perception? , 2007, Perception.

[44]  B. Velichkovsky,et al.  Eye typing in application: A comparison of two interfacing systems with ALS patients , 2008 .

[45]  Lester C. Loschky,et al.  What Would Jaws Do? The Tyranny of Film and the Relationship between Gaze and Higher-Level Narrative Film Comprehension , 2015, PloS one.

[46]  J. C. Stubbs.  The Evolution of Orson Welles's "Touch of Evil" from Novel to Film , 1985 .

[47]  B. Scholl,et al.  Cognition does not affect perception: Evaluating the evidence for “top-down” effects , 2015, Behavioral and Brain Sciences.

[48]  Miguel P Eckstein,et al.  Attentional Cues in Real Scenes, Saccadic Targeting, and Bayesian Priors , 2005, Psychological science.

[49]  Marianne A. DeAngelus,et al.  Top-down control of eye movements: Yarbus revisited , 2009 .

[50]  Joan E. Hart,et al.  Film Directing Shot by Shot: Visualizing from Concept to Screen , 1991 .

[51]  Jeffrey N. Rouder,et al.  Default Bayes factors for ANOVA designs , 2012 .

[52]  Daniel L. Dinsmore,et al.  “Reading” Paintings: Evidence for Trans-Symbolic and Symbol-Specific Comprehension Processes , 2015 .

[53]  T. Trabasso,et al.  Constructing inferences during narrative text comprehension. , 1994, Psychological review.

[54]  Richard J. Gerrig,et al.  Neural evidence that suspense narrows attentional focus , 2015, Neuroscience.

[55]  Jukka Hyönä,et al.  Processing of Written Irony: An Eye Movement Study , 2014 .

[56]  Jan Theeuwes,et al.  ScanMatch: A novel method for comparing fixation sequences , 2010, Behavior research methods.

[57]  K. Rayner,et al.  Eye Movements as Reflections of Comprehension Processes in Reading , 2006 .

[58]  Tad T. Brunyé,et al.  Gardony Map Drawing Analyzer: Software for quantitative analysis of sketch maps , 2016, Behavior research methods.

[59]  Richard C. Anderson,et al.  Recall of Previously Unrecallable Information Following a Shift in Perspective. Technical Report No. 41. , 1977 .

[60]  W. Kintsch,et al.  Strategies of discourse comprehension , 1983 .

[61]  Nida Latif,et al.  The art of gaze guidance. , 2014, Journal of experimental psychology. Human perception and performance.

[62]  Arthur C. Graesser,et al.  A three-pronged method for studying inference generation in literary text , 1991 .

[63]  Michael L. Mack,et al.  VISUAL SALIENCY DOES NOT ACCOUNT FOR EYE MOVEMENTS DURING VISUAL SEARCH IN REAL-WORLD SCENES , 2007 .

[64]  J. Bateman,et al.  Revisiting Cinematic Authorship : A Multimodal Approach , 2013 .

[65]  J. Henderson,et al.  Where Science and Magic Meet: The Illusion of a “Science of Magic” , 2010 .

[66]  D. S. Wooding,et al.  Fixation sequences made during visual examination of briefly presented 2D images. , 1997, Spatial vision.

[67]  Susanne Trauzettel-Klosinski,et al.  From Eye to Mind: Information Acquisition in Perception, Search, and Reading , 1997 .

[68]  K. Rayner,et al.  Effects of titles on the processing of text and lexically ambiguous words: Evidence from eye movements , 2000, Memory & cognition.

[69]  A. Coutrot,et al.  How saliency, faces, and sound influence gaze in dynamic social scenes. , 2014, Journal of vision.

[70]  Jeffrey N. Rouder,et al.  Bayesian t tests for accepting and rejecting the null hypothesis , 2009, Psychonomic bulletin & review.

[71]  T. Smith.  The attentional theory of cinematic continuity , 2012 .

[72]  Karel Reisz.  The Technique of Film Editing , 1957 .

[73]  John M. Henderson,et al.  Predicting Cognitive State from Eye Movements , 2013, PloS one.

[74]  M. Chun,et al.  Contextual Cueing: Implicit Learning and Memory of Visual Context Guides Spatial Attention , 1998, Cognitive Psychology.

[75]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[76]  John M. Henderson,et al.  Clustering of Gaze During Dynamic Scene Viewing is Predicted by Motion , 2011, Cognitive Computation.

[77]  Morton Ann Gernsbacher,et al.  Language Comprehension As Structure Building , 1990 .

[78]  T. Smith,et al.  Attentional synchrony and the influence of viewing task on gaze behavior in static and dynamic scenes. , 2013, Journal of vision.

[79]  J. Bittner,et al.  Cognitive Research : Principles and Implications , 2017 .

[80]  Jukka Hyönä,et al.  Effects of Topic Headings on Text Processing: Evidence from Adult Readers' Eye Fixation Patterns. , 2004 .

[81]  Rolf A. Zwaan,et al.  Indexing space and time in film understanding , 2001 .

[82]  L. Parra,et al.  Human Neuroscience Original Research Article Correlated Components of Ongoing Eeg Point to Emotionally Laden Attention – a Possible Marker of Engagement? , 2022 .

[83]  D. Simons,et al.  Failure to detect changes to people during a real-world interaction , 1998 .

[84]  Marcia K. Johnson,et al.  Contextual prerequisites for understanding: Some investigations of comprehension and recall , 1972 .

[85]  P. Thorndyke.  Cognitive structures in comprehension and memory of narrative discourse , 1977, Cognitive Psychology.

[86]  D. Heeger,et al.  Neurocinematics: The Neuroscience of Film , 2008 .

[87]  L. Itti.  Author address: , 1999 .

[88]  Lester C. Loschky,et al.  CHAPTER 8 Is Reading the Same as Viewing ? An Exploration of the Similarities and Differences Between Processing Text-and Visually Based Narratives , 2013 .

[89]  Jeffrey M. Zacks,et al.  Event structure in perception and conception. , 2001, Psychological bulletin.

[90]  Kristian Tylén,et al.  Brains striving for coherence: Long-term cumulative plot formation in the default mode network , 2015, NeuroImage.

[91]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[92]  R. Buckner,et al.  Functional-Anatomic Fractionation of the Brain's Default Network , 2010, Neuron.

[93]  Kaitlin L. Brunick,et al.  Quicker, faster, darker: Changes in Hollywood film over 75 years , 2011, i-Perception.

[94]  M. Gernsbacher,et al.  Investigating differences in general comprehension skill. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[95]  C. M. Mayo.  A Touch of Evil , 2002 .

[96]  E. Reingold,et al.  Saccadic Inhibition and Gaze Contingent Research Paradigms , 2000 .

[97]  Daniel T. Levin,et al.  A Window on Reality , 2012 .

[98]  J. Henderson.  Regarding Scenes , 2007 .

[99]  Stuart Hall,et al.  Encoding and Decoding in the television discourse , 1973 .

[100]  J. Henderson,et al.  Do the eyes really have it? Dynamic allocation of attention when viewing moving faces. , 2012, Journal of vision.

[101]  Annette Michelson,et al.  What is Cinema? , 1995 .

[102]  David Bordwell,et al.  Film Art: An Introduction , 1979 .

[103]  K. Rayner,et al.  Pronoun assignment and semantic integration during reading: eye movements and immediacy of processing , 1983 .

[104]  W. Murch.  In the blink of an eye : a perspective on film editing , 2001 .

[105]  W. Kintsch.  The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[106]  L. Itti,et al.  Mechanisms of top-down attention , 2011, Trends in Neurosciences.

[107]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[108]  J. Henderson,et al.  How are eye fixation durations controlled during scene viewing? Further evidence from a scene onset delay paradigm , 2009 .

[109]  Thinking in Pictures: Dramatic Structure in D. W. Griffith's Biograph Films , 1987 .

[110]  Jason E. Albrecht,et al.  Comprehension strategies in the development of a mental model. , 1992, Journal of experimental psychology. Learning, memory, and cognition.

[111]  R. Ratcliff,et al.  Inference during reading. , 1992, Psychological review.

[112]  T. Smith.  The role of audience participation and task relevance on change detection during a card trick , 2015, Front. Psychol..

[113]  J. Cutting.  The Framing of Characters in Popular Movies , 2015 .

[114]  T. Foulsham,et al.  Speaking and Listening with the Eyes: Gaze Signaling during Dyadic Interactions , 2015, PloS one.

[115]  Rolf A. Zwaan,et al.  The Construction of Situation Models in Narrative Comprehension: An Event-Indexing Model , 1995 .

[116]  J. Cutting.  The Reality of Illusion: An Ecological Approach to Cognitive Film Theory , 1999 .

[117]  R. Gregory.  The intelligent eye , 1970 .

[118]  Antonio Torralba,et al.  Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. , 2006, Psychological review.

[119]  L. Itti,et al.  Visual causes versus correlates of attentional selection in dynamic scenes , 2006, Vision Research.

[120]  Erik D. Reichle,et al.  Toward a model of eye movement control in reading. , 1998, Psychological review.

[121]  Krista A. Ehinger,et al.  Modelling search for people in 900 scenes: A combined source model of eye guidance , 2009 .