Eye guidance in natural vision: reinterpreting salience.

Models of gaze allocation in complex scenes are derived mainly from studies of static picture viewing. The dominant framework to emerge has been image salience, where properties of the stimulus play a crucial role in guiding the eyes. However, salience-based schemes are poor at accounting for many aspects of picture viewing and can fail dramatically in the context of natural task performance. These failures have led to the development of new models of gaze allocation in scene viewing that address a number of these issues. However, models based on the picture-viewing paradigm are unlikely to generalize to a broader range of experimental contexts, because the stimulus context is limited, and the dynamic, task-driven nature of vision is not represented. We argue that there is a need to move away from this class of model and find the principles that govern gaze allocation in a broader range of settings. We outline the major limitations of salience-based selection schemes and highlight what we have learned from studies of gaze allocation in natural vision. Clear principles of selection are found across many instances of natural vision and these are not the principles that might be expected from picture-viewing studies. We discuss the emerging theoretical framework for gaze allocation on the basis of reward maximization and uncertainty reduction.

[1]  R. C. Langford.  How People Look at Pictures, A Study of the Psychology of Perception in Art. , 1936 .

[2]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[3]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[4]  W. Becker.  The control of eye movements in the saccadic system. , 1972, Bibliotheca ophthalmologica : supplementa ad ophthalmologica.

[5]  L. Stark,et al.  Most naturally occurring human saccades have magnitudes of 15 degrees or less. , 1975, Investigative ophthalmology.

[6]  M. Jeannerod,et al.  Corrective saccades: dependence on retinal reafferent signals , 1975, Vision Research.

[7]  G. K. Noorden.  Movements of the Eyes , 1978 .

[8]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[9]  D. Bouwhuis,et al.  Attention and performance X : control of language processes , 1986 .

[10]  W. Wolf,et al.  The Evaluation of the Oculomotor Error Signal , 1984 .

[11]  M. Posner,et al.  Components of visual orienting , 1984 .

[12]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[13]  J. Stern.  Theoretical and applied aspects of eye movement research A. G. Gale and F. Johnson, (Elsevier Science Publishers B.V., Amsterdam, 1984) pp. xiii + 565, Dfl. 185 , 1985, Biological Psychology.

[14]  D. Robinson,et al.  Saccadic undershoot is not inevitable: Saccades can be accurate , 1986, Vision Research.

[15]  Carolina M. Zingale,et al.  Planning sequences of saccades , 1987, Vision Research.

[16]  Eileen Kowler,et al.  The role of location probability in the programming of saccades: Implications for “center-of-gravity” tendencies , 1989, Vision Research.

[17]  C. Bundesen.  A theory of visual attention. , 1990, Psychological review.

[18]  Eileen Kowler.  Eye movements and their role in visual and cognitive processes. , 1990, Reviews of oculomotor research.

[19]  Eileen Kowler.  The role of visual and cognitive processes in the control of eye movement. , 1990, Reviews of oculomotor research.

[20]  S. Petersen,et al.  The pulvinar and visual salience , 1992, Trends in Neurosciences.

[21]  O. Mimura.  [Eye movements]. , 1992, Nippon Ganka Gakkai zasshi.

[22]  R. Klein,et al.  Does Oculomotor Readiness Mediate Cognitive Control of Visual-Attention - Revisited , 1994 .

[23]  David N. Lee,et al.  Where we look when we steer , 1994, Nature.

[24]  H. Collewijn,et al.  The function of visual search and memory in sequential looking tasks , 1995, Vision Research.

[25]  W. Schneider.  VAM: A neuro-cognitive model for visual attention control of segmentation, object recognition, and space-based motor action , 1995 .

[26]  H. Deubel,et al.  Saccade target selection and object recognition: Evidence for a common attentional mechanism , 1996, Vision Research.

[27]  M F Land,et al.  The knowledge base of the oculomotor system. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[28]  Rajesh P. N. Rao,et al.  PSYCHOLOGICAL SCIENCE Research Article EYE MOVEMENTS REVEAL THE SPATIOTEMPORAL DYNAMICS OE VISUAL SEARCH , 2022 .

[29]  D. S. Wooding,et al.  Fixation sequences made during visual examination of briefly presented 2D images. , 1997, Spatial vision.

[30]  A E Patla,et al.  Where and when do we look as we approach and step over an obstacle in the travel path? , 1997, Neuroreport.

[31]  D. Ballard,et al.  Task constraints in visual working memory , 1997, Vision Research.

[32]  K. Hoffmann,et al.  Optokinetic eye movements elicited by radial optic flow in the macaque monkey. , 1998, Journal of neurophysiology.

[33]  L. P. O'Keefe,et al.  The influence of fixational eye movements on the response of neurons in area MT of the macaque , 1998, Visual Neuroscience.

[34]  J. Wolfe,et al.  What Can 1 Million Trials Tell Us About Visual Search? , 1998 .

[35]  M. Goldberg,et al.  The representation of visual salience in monkey parietal cortex , 1998, Nature.

[36]  P. Chapman,et al.  Visual Search of Driving Situations: Danger and Experience , 1998, Perception.

[37]  M. Land,et al.  The Roles of Vision and Eye Movements in the Control of Activities of Daily Living , 1998, Perception.

[38]  M. Land,et al.  The effects of skill on the eye–hand span during musical sight–reading , 1999, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[39]  P Reinagel,et al.  Natural scene statistics at the centre of gaze. , 1999, Network.

[40]  Michael L. Platt,et al.  Neural correlates of decision variables in parietal cortex , 1999, Nature.

[41]  N. P. Bichot,et al.  Effects of similarity and history on neural mechanisms of visual selection , 1999, Nature Neuroscience.

[42]  R. Klein,et al.  Inhibition of Return is a Foraging Facilitator in Visual Search , 1999 .

[43]  J. Schall,et al.  Performance monitoring by the supplementary eye ® eld , 2000 .

[44]  Michael F. Land,et al.  From eye movements to actions: how batsmen hit the ball , 2000, Nature Neuroscience.

[45]  D. E. Irwin,et al.  Attentional and oculomotor capture by onset, luminance and color singletons , 2000, Vision Research.

[46]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[47]  J. Hollerman,et al.  Reward processing in primate orbitofrontal cortex and basal ganglia. , 2000, Cerebral cortex.

[48]  W. Schultz.  Multiple reward signals in the brain , 2000, Nature Reviews Neuroscience.

[49]  M. Hayhoe,et al.  What controls attention in natural environments? , 2001, Vision Research.

[50]  J. Theeuwes,et al.  Attentional and oculomotor capture , 2001 .

[51]  B. Tatler,et al.  Steering with the head The visual strategy of a racing driver , 2001, Current Biology.

[52]  R. Carpenter,et al.  Express saccades: is bimodality a result of the order of stimulus presentation? , 2001, Vision Research.

[53]  J. Pelz,et al.  Oculomotor behavior and perceptual strategies in complex tasks , 2001, Vision Research.

[54]  R. Johansson,et al.  Eye–Hand Coordination in Object Manipulation , 2001, The Journal of Neuroscience.

[55]  B. Gibson,et al.  Attraction, Distraction and Action: Multiple Perspectives on Attentional Capture. Advances in Psychology , 2001 .

[56]  B. Hood,et al.  Is Visual Search Really like Foraging? , 2001, Perception.

[57]  Zhaoping Li.  A saliency map in primary visual cortex , 2002, Trends in Cognitive Sciences.

[58]  N. Badler,et al.  Eyes Alive Eyes Alive Eyes Alive Figure 1: Sample Images of an Animated Face with Eye Movements , 2022 .

[59]  Robert M. McPeek,et al.  Superior colliculus activity related to concurrent processing of saccade goals in a visual search task. , 2002, Journal of neurophysiology.

[60]  Joan N. Vickers,et al.  How far ahead do we look when required to step on specific locations in the travel path during locomotion? , 2002, Experimental Brain Research.

[61]  Derrick J. Parkhurst,et al.  Modeling the role of salience in the allocation of overt visual attention , 2002, Vision Research.

[62]  D. Simons,et al.  Moving and looming stimuli capture attention , 2003, Perception & psychophysics.

[63]  K. Turano,et al.  Oculomotor strategies for the direction of gaze tested with a real-world activity , 2003, Vision Research.

[64]  Mary M Hayhoe,et al.  Visual memory and motor planning in a natural task. , 2003, Journal of vision.

[65]  Michael S Landy,et al.  Statistical decision theory and the selection of rapid, goal-directed movements. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[66]  J. Gallant,et al.  Goal-Related Activity in V4 during Free Viewing Visual Search Evidence for a Ventral Stream Visual Salience Map , 2003, Neuron.

[67]  P. König,et al.  Does luminance‐contrast contribute to a saliency map for overt visual attention? , 2003, The European journal of neuroscience.

[68]  P. Glimcher.  The neurobiology of visual-saccadic decision making. , 2003, Annual review of neuroscience.

[69]  Michael F. Land,et al.  The coordination of rotations of the eyes, head and trunk in saccadic turns produced in natural situations , 2004, Experimental Brain Research.

[70]  Jonathan D. Cohen,et al.  Computational roles for dopamine in behavioural control , 2004, Nature.

[71]  P. König,et al.  Interactions between eye movement systems in cats and humans , 2004, Experimental Brain Research.

[72]  Cristina Massen,et al.  Parallel programming of exogenous and endogenous components in the antisaccade task , 2004, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[73]  P. Glimcher,et al.  Activity in Posterior Parietal Cortex Is Correlated with the Relative Subjective Desirability of Action , 2004, Neuron.

[74]  W. Newsome,et al.  Matching Behavior and the Representation of Value in the Parietal Cortex , 2004, Science.

[75]  Jitendra Malik,et al.  An Information Maximization Model of Eye Movements , 2004, NIPS.

[76]  Michael L. Mack,et al.  Human Gaze Control in RealWorld Search , 2004, WAPCV.

[77]  J. Henderson,et al.  Prioritization of new objects in real-world scenes: evidence from eye movements. , 2005, Journal of experimental psychology. Human perception and performance.

[78]  Wilson S. Geisler,et al.  Optimal eye movement strategies in visual search , 2005, Nature.

[79]  L. Itti,et al.  Modeling the influence of task on attention , 2005, Vision Research.

[80]  L. Itti.  Author address: , 1999 .

[81]  I. Hooge,et al.  Inhibition of return is not a foraging facilitator in saccadic search and free viewing , 2005, Vision Research.

[82]  G. Kuhn,et al.  Magic and Fixation: Now You Don't See it, Now You Do , 2005, Perception.

[83]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[84]  N. P. Bichot,et al.  A visual salience map in the primate frontal eye field. , 2005, Progress in brain research.

[85]  Asha Iyer,et al.  Components of bottom-up gaze allocation in natural images , 2005, Vision Research.

[86]  Jason A. Droll,et al.  Task demands control acquisition and storage of visual information. , 2005, Journal of experimental psychology. Human perception and performance.

[87]  Iain D. Gilchrist,et al.  Visual correlates of fixation selection: effects of scale and time , 2005, Vision Research.

[88]  R. Johansson,et al.  Eye–Hand Coordination during Learning of a Novel Visuomotor Task , 2005, The Journal of Neuroscience.

[89]  Michael L. Platt,et al.  Monkeys Pay Per View: Adaptive Valuation of Social Images by Rhesus Macaques , 2005, Current Biology.

[90]  Jillian H. Fecteau,et al.  Salience, relevance, and firing: a priority map for target selection , 2006, Trends in Cognitive Sciences.

[91]  R. Baddeley,et al.  The long and the short of it: Spatial statistics at fixation vary with saccade amplitude and task , 2006, Vision Research.

[92]  Kristin C. Scott,et al.  A Heterochromatin Barrier Partitions the Fission Yeast Centromere into Discrete Chromatin Domains , 2006, Current Biology.

[93]  Kae Nakamura,et al.  Basal ganglia orient eyes to reward. , 2006, Journal of neurophysiology.

[94]  Michael S Landy,et al.  Combining Priors and Noisy Visual Cues in a Rapid Pointing Task , 2006, The Journal of Neuroscience.

[95]  M. Land.  Eye movements and the control of actions in everyday life , 2006, Progress in Retinal and Eye Research.

[96]  Mary Hayhoe,et al.  Control of attention and gaze in complex environments. , 2006, Journal of vision.

[97]  Stephen V. Shepherd,et al.  Social status gates social attention in monkeys , 2006, Current Biology.

[98]  T. Foulsham,et al.  Eye movements during scene inspection: A test of the saliency map hypothesis , 2006 .

[99]  Antonio Torralba,et al.  Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. , 2006, Psychological review.

[100]  Christof Koch,et al.  Modeling attention to salient proto-objects , 2006, Neural Networks.

[101]  I. Gilchrist,et al.  Evidence for a systematic component within scan paths in visual search , 2006 .

[102]  L. Itti,et al.  Visual causes versus correlates of attentional selection in dynamic scenes , 2006, Vision Research.

[103]  J. Schall,et al.  Executive control of countermanding saccades by the supplementary eye field , 2006, Nature Neuroscience.

[104]  Preeti Verghese,et al.  Where to look next? Eye movements reduce local uncertainty. , 2007, Journal of vision.

[105]  Jeremy M. Wolfe,et al.  Guided Search 4.0: Current Progress With a Model of Visual Search , 2007, Integrated Models of Cognitive Systems.

[106]  Frank E. Ritter,et al.  The Rise of Cognitive Architectures , 2007, Integrated Models of Cognitive Systems.

[107]  Iain D. Gilchrist,et al.  Investigating a space-variant weighted salience account of visual selection , 2007, Vision Research.

[108]  Dana H. Ballard,et al.  Modeling embodied visual behaviors , 2007, TAP.

[109]  Jason A. Droll,et al.  Learning where to direct gaze during change detection. , 2007, Journal of vision.

[110]  J. Henderson.  Regarding Scenes , 2007 .

[111]  Mary M Hayhoe,et al.  Task and context determine where you look. , 2016, Journal of vision.

[112]  Benjamin W Tatler,et al.  The central fixation bias in scene viewing: selecting an optimal viewing position independently of motor biases and image feature distributions. , 2007, Journal of vision.

[113]  Michael L. Mack,et al.  VISUAL SALIENCY DOES NOT ACCOUNT FOR EYE MOVEMENTS DURING VISUAL SEARCH IN REAL-WORLD SCENES , 2007 .

[114]  Gustav Kuhn,et al.  DON'T LOOK NOW: THE MAGIC OF MISDIRECTION , 2007 .

[115]  Guy Thomas Buswell,et al.  An experimental study of the eye-voice span in reading , 2008 .

[116]  John K. Tsotsos,et al.  Attention and Performance in Computational Vision , 2008 .

[117]  Benjamin W. Tatler,et al.  Systematic tendencies in scene viewing , 2008 .

[118]  Mary M. Hayhoe,et al.  Is attention drawn to changes in familiar scenes? , 2008 .

[119]  Marcus Nyström,et al.  Semantic override of low-level features in image viewing - both initially and overall , 2008 .

[120]  M. Jeasnerod.  CORRECTIVE SACCADES : DEPENDENCE ON RETINAL REAFFERENT SIGNALS , 2008 .

[121]  Steven Franconeri,et al.  Objects on a Collision Path With the Observer Demand Attention , 2008, Psychological science.

[122]  Raymond Klein,et al.  Inhibition of return , 2000, Trends in Cognitive Sciences.

[123]  J. Henderson,et al.  Prioritizing new objects for eye fixation in real-world scenes: Effects of object–scene consistency , 2008 .

[124]  Jiri Najemnik,et al.  Eye movement statistics in humans are consistent with an optimal search strategy. , 2008, Journal of vision.

[125]  C. Koch,et al.  Task-demands can immediately reverse the effects of sensory-driven saliency in complex visual stimuli. , 2008, Journal of vision.

[126]  M. Landy,et al.  Decision making, movement planning and statistical decision theory , 2008, Trends in Cognitive Sciences.

[127]  P. Perona,et al.  Objects predict fixations better than early saliency. , 2008, Journal of vision.

[128]  T. Foulsham,et al.  What can saliency models predict about eye movements? Spatial and sequential aspects of fixations during encoding and recognition. , 2008, Journal of vision.

[129]  Colin Camerer,et al.  Neuroeconomics: decision making and the brain , 2008 .

[130]  Graham L. Pierce,et al.  Eye movements during scene viewing: Evidence for mixed control of fixation durations , 2008, Psychonomic bulletin & review.

[131]  Brian C. McCann,et al.  Learning Stochastic Reward Distributions in a Speeded Pointing Task , 2008, The Journal of Neuroscience.

[132]  G. Zelinsky.  A theory of eye movements during target acquisition. , 2008, Psychological review.

[133]  B. Tatler,et al.  The prominence of behavioural biases in eye guidance , 2009 .

[134]  J. Henderson,et al.  Facilitation of return during scene viewing , 2009 .

[135]  Jochen J. Steil,et al.  Integrating Inhomogeneous Processing and Proto-object Formation in a Computational Model of Visual Attention , 2009, Human Centered Robot Systems, Cognition, Interaction, Technology.

[136]  G. Kuhn,et al.  You look where I look! Effect of gaze cues on overt and covert attention in misdirection , 2009 .

[137]  George L. Malcolm,et al.  Searching in the dark: Cognitive relevance drives attention in real-world scenes , 2009, Psychonomic bulletin & review.

[138]  John K. Tsotsos,et al.  Saliency, attention, and visual search: an information theoretic approach. , 2009, Journal of vision.

[139]  Roland J. Baddeley,et al.  The nature of the visual representations involved in eye movements when walking down the street , 2009 .

[140]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[141]  Simon Farrell,et al.  Influence of environmental statistics on inhibition of saccadic return , 2009, Proceedings of the National Academy of Sciences.

[142]  P. Glimcher,et al.  Visual processing, learning and feedback in the primate eye movement system , 2009, Trends in Neurosciences.

[143]  P. König,et al.  Gaze allocation in natural stimuli: Comparing free exploration to head-fixed viewing conditions , 2009 .

[144]  R. Baddeley,et al.  Do we look at lights? Using mixture modelling to distinguish between low- and high-level factors in natural image viewing , 2009 .

[145]  D. Ballard,et al.  Modelling the role of task in the control of gaze , 2009, Visual cognition.

[146]  B. Tatler,et al.  Looking and Acting: Vision and eye movements in natural behaviour , 2009 .

[147]  Karl R Gegenfurtner,et al.  Effects of salience and reward information during saccadic decisions under risk. , 2009, Journal of the Optical Society of America. A, Optics, image science, and vision.

[148]  M. Hayhoe,et al.  Adaptive Gaze Control in Natural Environments , 2009, The Journal of Neuroscience.

[149]  Pierre Baldi,et al.  Bayesian surprise attracts human attention , 2005, Vision Research.

[150]  J. Henderson,et al.  Overt attentional prioritization of new objects and feature changes during real-world scene viewing , 2009 .

[151]  Matthew H Tong,et al.  SUN: Top-down saliency using natural statistics , 2009, Visual cognition.

[152]  J. Henderson,et al.  How are eye fixation durations controlled during scene viewing? Further evidence from a scene onset delay paradigm , 2009 .

[153]  Krista A. Ehinger,et al.  Modelling search for people in 900 scenes: A combined source model of eye guidance , 2009 .

[154]  Constantin A Rothkopf,et al.  Image statistics at the point of gaze during human navigation , 2009, Visual Neuroscience.

[155]  Dana H. Ballard,et al.  Credit Assignment in Multiple Goal Embodied Visuomotor Behavior , 2010, Front. Psychology.

[156]  Thomas Martinetz,et al.  Variability of eye movements when viewing dynamic natural scenes. , 2010, Journal of vision.

[157]  Alan Kennedy,et al.  Perception and memory across viewpoint changes in moving images. , 2010, Journal of vision.

[158]  Jinhua Xu,et al.  Emergence of Visual Saliency from Natural Scenes via Context-Mediated Probability Distributions Coding , 2010, PloS one.

[159]  Jan-Mark Geusebroek,et al.  An Image Statistics–Based Model for Fixation Prediction , 2010, Cognitive Computation.

[160]  Jochen J. Steil,et al.  Where to Look Next? Combining Static and Dynamic Proto-objects in a TVA-based Model of Visual Attention , 2010, Cognitive Computation.

[161]  Mary Hayhoe,et al.  Visual Cognition , 2021, Encyclopedia of Evolutionary Psychological Science.

[162]  J. Henderson,et al.  CRISP: a computational model of fixation durations in scene viewing. , 2010, Psychological review.

[163]  Donald Morrill.  SACCADES , 2010, Fourth Genre: Explorations in Nonfiction.

[164]  Pietro Perona,et al.  Optimal reward harvesting in complex perceptual environments , 2010, Proceedings of the National Academy of Sciences.

[165]  P. Cavanagh.  Visual cognition , 2011, Vision Research.

[166]  H. Müller,et al.  Visually guided pointing movements are driven by the salience map. , 2011, Journal of vision.

[167]  John G. Taylor,et al.  Saliency, Attention, Active Visual Search, and Picture Scanning , 2011, Cognitive Computation.

[168]  Christof Koch,et al.  Learning a saliency map using fixated locations in natural scenes. , 2011, Journal of vision.

[169]  E. Benarroch.  Executive Control , 2022, Neuroscience for Clinicians.