Attention, Movie Cuts, and Natural Vision: A Functional Perspective

Paying attention to the right thing at the right time underlies the ability of humans and other animals to learn, perceive, and interact with their environment. Attentional selection enables biological organisms with limited neural resources to extract the most pertinent information from the barrage of sensory inputs that they normally experience in the real world. The goal of the research described in this dissertation is to characterize the output, input, and associated computational transformations that underlie attentional selection during natural vision. Such functional understanding of attentional selection would help build more intelligent machines that could behave autonomously in real world environments. It could also lead to better diagnostic tools and treatments for medical conditions, such as Autism, ADHD, and Parkinson Disease, which are characterized by anomalous patterns of attentional selection. The following key questions are addressed: What do movies reveal about the nature of sensory inputs to the human visual system? Which sensory inputs causally attract attention and how do they compare to each other? How is attentional selection affected by interactions between past and present sensory inputs? To answer these questions, the eyes of human observers were tracked non-intrusively as they watched either continuous or MTV-style (discontinuous) video clips. The main conclusions of this dissertation are based on a series of quantitative analyses, in which human gaze behavior and the prediction accuracy of related computational models were compared across viewing conditions, space, and time. The results indicate that sensory inputs are naturally discontinuous, and that the human visual system can keep the mind informed of the most pertinent information at every point in time without requiring either visual or mental continuity. It is also shown that dynamic visual correlates of attentional selection (e.g., motion contrasts) play a dominant causal role in attracting attention during natural vision. In comparison, some static visual correlates (e.g., color contrasts) play a relatively weaker causal role, whereas others (e.g., orientation contrast) are non causal correlates, potentially reflecting top-down causes. Lastly, it is demonstrated that perceptual memory is utilized for guiding attention across several gaze shits when persistent visual context is available.

[1]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[2]  S. Yantis,et al.  Visual motion and attentional capture , 1994, Perception & psychophysics.

[3]  David J. Field,et al.  How Close Are We to Understanding V1? , 2005, Neural Computation.

[4]  Asha Iyer,et al.  Components of bottom-up gaze allocation in natural images , 2005, Vision Research.

[5]  Michael F. Land,et al.  From eye movements to actions: how batsmen hit the ball , 2000, Nature Neuroscience.

[6]  Wa James Tam,et al.  Static and dynamic spatial resolution in image coding: an investigation of eye movements , 1991, Electronic Imaging.

[7]  J. Gibson.  The Ecological Approach to Visual Perception , 1979 .

[8]  Derrick J. Parkhurst,et al.  Modeling the role of salience in the allocation of overt visual attention , 2002, Vision Research.

[9]  G. Hauske,et al.  Object and scene analysis by saccadic eye-movements: an investigation with higher-order statistics. , 2000, Spatial vision.

[10]  J. O'Regan,et al.  Solving the "real" mysteries of visual perception: the world as an outside memory. , 1992, Canadian journal of psychology.

[11]  K. Nakayama,et al.  On the Functional Role of Implicit Visual Memory for the Adaptive Deployment of Attention Across Scenes , 2000 .

[12]  J. Henderson.  Human gaze control during real-world scene perception , 2003, Trends in Cognitive Sciences.

[13]  E. Reed.  The Ecological Approach to Visual Perception , 1989 .

[14]  R. Hernández-Peón,et al.  Modification of electric activity in cochlear nucleus during attention in unanesthetized cats. , 1956, Science.

[15]  Todd S. Horowitz,et al.  Visual search has no memory , 1998, Nature.

[16]  Iain D. Gilchrist,et al.  Visual correlates of fixation selection: effects of scale and time , 2005, Vision Research.

[17]  J. Atkinson,et al.  Neurobiological models of normal and abnormal visual development , 2005 .

[18]  Paul Messaris,et al.  Visual ""literacy"": Image, Mind, And Reality , 1994 .

[19]  Laurent Itti,et al.  The role of memory in guiding attention during natural vision. , 2006, Journal of vision.

[20]  Steven R. Holloway,et al.  Visual experience can substantially alter critical flicker fusion thresholds , 2005, Human psychopharmacology.

[21]  Eileen Kowler,et al.  Visual scene memory and the guidance of saccadic eye movements , 2001, Vision Research.

[22]  Karel Reisz.  The Technique of Film Editing , 1957 .

[23]  Prof. Dr. Eberhard Curio.  The Ethology of Predation , 1976, Zoophysiology and Ecology.

[24]  P. Tse,et al.  Time and the Brain: How Subjective Time Relates to Neural Time , 2005 .

[25]  Ronald A. Rensink.  Seeing, sensing, and scrutinizing , 2000, Vision Research.

[26]  David L. Sheinberg,et al.  Noticing Familiar Objects in Real World Scenes: The Role of Temporal Cortical Neurons in Natural Vision , 2001, The Journal of Neuroscience.

[27]  U Polat,et al.  Spatial interactions in human vision: from near to far via experience-dependent cascades of connections. , 1994, Proceedings of the National Academy of Sciences of the United States of America.

[28]  James E. Cutting,et al.  Perceiving Scenes in Film and in the World , 2001 .

[29]  M. Murray,et al.  EEG source imaging , 2004, Clinical Neurophysiology.

[30]  David Bordwell,et al.  Film History: An Introduction , 1994 .

[31]  K. Nakayama,et al.  Single visual neurons code opposing motion independent of direction. , 1983, Science.

[32]  S. Yantis,et al.  Uniqueness of abrupt visual onset in capturing attention , 1988, Perception & psychophysics.

[33]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[34]  K. Nakayama,et al.  Priming of pop-out: II. The role of position , 1996, Perception & psychophysics.

[35]  J. Cutting.  The Reality of Illusion: An Ecological Approach to Cognitive Film Theory , 1999 .

[36]  Julian Hochberg,et al.  Representation of motion and space in video and cinematic displays , 1986 .

[37]  Miguel P Eckstein,et al.  The time course of visual information accrual guiding eye movement decisions. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[38]  M. Land.  Motion and vision: why animals move their eyes , 1999, Journal of Comparative Physiology A.

[39]  D. Finlay,et al.  Cardiac and visual responses to moving stimuli presented either successively or simultaneously to the central and peripheral visual fields in 4-month-old infants. , 1984 .

[40]  D. V. van Essen,et al.  The representation of the visual field in parvicellular and magnocellular layers of the lateral geniculate nucleus in the macaque monkey , 1984, The Journal of comparative neurology.

[41]  Wilson S. Geisler,et al.  Optimal eye movement strategies in visual search , 2005, Nature.

[42]  Helga C. Arsenio,et al.  Panoramic search: the interaction of memory and vision in search through a familiar scene. , 2004, Journal of experimental psychology. Human perception and performance.

[43]  Gidon Felsen,et al.  A natural approach to studying vision , 2005, Nature Neuroscience.

[44]  Jillian H. Fecteau,et al.  Neural correlates of the automatic and goal-driven biases in orienting spatial attention. , 2004, Journal of neurophysiology.

[45]  R. Walker,et al.  A model of saccade generation based on parallel processing and competitive inhibition , 1999, Behavioral and Brain Sciences.

[46]  W. Geisler,et al.  Optimal Eye Movement Strategies in Visual Search ( Supplement ) , 2005 .

[47]  Antonio Torralba,et al.  Top-down control of visual attention in object detection , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[48]  S. Yantis,et al.  On the distinction between visual salience and stimulus-driven attentional capture. , 1999, Journal of experimental psychology. Human perception and performance.

[49]  K. Nakayama,et al.  Priming of pop-out: I. Role of features , 1994, Memory & cognition.

[50]  V. Tosi,et al.  Scanning eye movements made when viewing film: preliminary observations. , 1997, The International journal of neuroscience.

[51]  R. Malach,et al.  Intersubject Synchronization of Cortical Activity During Natural Vision , 2004, Science.

[52]  Nicole C. Rust,et al.  In praise of artifice , 2005, Nature Neuroscience.

[53]  J. C. Johnston,et al.  Involuntary covert orienting is contingent on attentional control settings. , 1992, Journal of experimental psychology. Human perception and performance.

[54]  M. Hayhoe,et al.  What controls attention in natural environments? , 2001, Vision Research.

[55]  F. Volkmar,et al.  Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. , 2002, Archives of general psychiatry.

[56]  David Melcher,et al.  Persistence of visual memory for scenes , 2001, Nature.

[57]  Terrence J. Sejnowski,et al.  Network model of shape-from-shading: neural function arises from both receptive and projective fields , 1988, Nature.

[58]  Janette Atkinson,et al.  Selection-for-Action and the Development of Orienting and Visual Attention , 1998 .

[59]  D. Simons,et al.  Failure to detect changes to attended objects in motion pictures , 1997 .

[60]  Derrick J. Parkhurst,et al.  Scene content selected by active vision. , 2003, Spatial vision.

[61]  L. Itti.  Author address: , 1999 .

[62]  J. Henderson,et al.  High-level scene perception. , 1999, Annual review of psychology.

[63]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[64]  M F Land,et al.  The knowledge base of the oculomotor system. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[65]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[66]  J. Wolfe,et al.  Attention is fast but volition is slow , 2000, Nature.

[67]  Klaus Bartl,et al.  A pivotable head mounted camera system that is aligned by three-dimensional eye movements , 2006, ETRA.

[68]  P. Viviani.  Eye movements in visual search: cognitive, perceptual and motor control aspects. , 1990, Reviews of oculomotor research.

[69]  Antonio Torralba,et al.  Modeling global scene factors in attention. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[70]  John M. Carroll,et al.  Toward A Structural Psychology Of Cinema , 1980 .

[71]  David N. Lee,et al.  Where we look when we steer , 1994, Nature.

[72]  D. S. Wooding,et al.  Fixation sequences made during visual examination of briefly presented 2D images. , 1997, Spatial vision.

[73]  P Reinagel,et al.  Natural scene statistics at the centre of gaze. , 1999, Network.

[74]  D. Spalding.  The Principles of Psychology , 1873, Nature.

[75]  Mary M Hayhoe,et al.  Visual memory and motor planning in a natural task. , 2003, Journal of vision.

[76]  Susan A. Murphy,et al.  Monographs on statistics and applied probability , 1990 .

[77]  R. Abrams,et al.  The onset of receding motion captures attention: Comment on Franconeri and Simons (2003) , 2005, Perception & psychophysics.

[78]  J. Mollon,et al.  Fruits, foliage and the evolution of primate colour vision. , 2001, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[79]  D. Robinson,et al.  Shared neural control of attentional shifts and eye movements , 1996, Nature.

[80]  J. Theeuwes.  Stimulus-driven capture and attentional set: selective search for color and visual abrupt onsets. , 1994, Journal of experimental psychology. Human perception and performance.

[81]  Raymond Klein,et al.  Inhibition of return , 2000, Trends in Cognitive Sciences.

[82]  D. S. Wooding,et al.  Fixation Patterns Made during Brief Examination of Two-Dimensional Images , 1997, Perception.

[83]  H. Jones,et al.  Visual cortical mechanisms detecting focal orientation discontinuities , 1995, Nature.

[84]  A. Hendrickson,et al.  Distribution of cones in human and monkey retina: individual variability and radial asymmetry. , 1987, Science.

[85]  D. Simons,et al.  Do New Objects Capture Attention? , 2005, Psychological science.

[86]  M. Chun,et al.  Contextual Cueing: Implicit Learning and Memory of Visual Context Guides Spatial Attention , 1998, Cognitive Psychology.

[87]  J. Henderson,et al.  Accurate visual memory for previously attended objects in natural scenes , 2002 .

[88]  L. Itti,et al.  Visual causes versus correlates of attentional selection in dynamic scenes , 2006, Vision Research.

[89]  David S Wooding,et al.  Eye movements of large populations: II. Deriving regions of interest, coverage, and similarity using fixation maps , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[90]  G. Sperling,et al.  Episodic theory of the dynamics of spatial attention. , 1995 .

[91]  M. Goldberg,et al.  The representation of visual salience in monkey parietal cortex , 1998, Nature.

[92]  P. Bex,et al.  Spatial frequency, phase, and the contrast of natural images. , 2002, Journal of the Optical Society of America. A, Optics, image science, and vision.