Complementary effects of gaze direction and early saliency in guiding fixations during free viewing.

Gaze direction provides an important and ubiquitous communication channel in daily behavior and social interaction of humans and some animals. While several studies have addressed gaze direction in synthesized simple scenes, few have examined how it can bias observer attention and how it might interact with early saliency during free viewing of natural and realistic scenes. Experiment 1 used a controlled, staged setting in which an actor was asked to look at two different objects in turn, yielding two images that differed only by the actor's gaze direction, to causally assess the effects of actor gaze direction. Over all scenes, the median probability of following an actor's gaze direction was higher than the median probability of looking toward the single most salient location, and higher than chance. Experiment 2 confirmed these findings over a larger set of unconstrained scenes collected from the Web and containing people looking at objects and/or other people. To further compare the strength of saliency versus gaze direction cues, we computed gaze maps by drawing a cone in the direction of gaze of the actors present in the images. Gaze maps predicted observers' fixation locations significantly above chance, although below saliency. Finally, to gauge the relative importance of actor face and eye directions in guiding observer's fixations, in Experiment 3, observers were asked to guess the gaze direction from only an actor's face region (with the rest of the scene masked), in two conditions: actor eyes visible or masked. Median probability of guessing the true gaze direction within ±9° was significantly higher when eyes were visible, suggesting that the eyes contribute significantly to gaze estimation, in addition to face region. Our results highlight that gaze direction is a strong attentional cue in guiding eye movements, complementing low-level saliency cues, and derived from both face and eyes of actors in the scene. Thus gaze direction should be considered in constructing more predictive visual attention models in the future.

[1]  J. Gibson,et al.  Perception of another person's looking behavior. , 1963, The American journal of psychology.

[2]  U. Neisser.  VISUAL SEARCH. , 1964, Scientific American.

[3]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[4]  M. Cline.  The perception of where a person is looking. , 1967, The American journal of psychology.

[5]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[6]  D. Kupfer,et al.  Interval between onset of sleep and rapid-eye-movement sleep as an indicator of depression. , 1972, Lancet.

[7]  J. Bruner,et al.  The capacity for joint visual attention in the infant , 1975, Nature.

[8]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[9]  M. Posner,et al.  Orienting of Attention* , 1980, The Quarterly journal of experimental psychology.

[10]  L. Adamson,et al.  Coordinating attention to people and objects in mother-infant and peer-infant interaction. , 1984, Child development.

[11]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[12]  G. Butterworth,et al.  What minds have in common is space : Spatial mechanisms serving joint visual attention in infancy , 1991 .

[13]  A. Gopnik,et al.  Natural theories of mind: Evolution, development and simulation of everyday mindreading , 2015 .

[14]  G. Butterworth.  The ontogeny and phylogeny of joint visual attention. , 1991 .

[15]  Thierry Pun,et al.  Integration of bottom-up and top-down cues for visual attention using non-linear relaxation , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[16]  David N. Lee,et al.  Where we look when we steer , 1994, Nature.

[17]  Dare A. Baldwin,et al.  Understanding the link between joint attention and language. , 1995 .

[18]  A. Karmiloff-Smith,et al.  Are children with autism blind to the mentalistic significance of the eyes , 1995 .

[19]  C. Moore,et al.  Joint attention : its origins and role in development , 1995 .

[20]  M. V. von Grünau,et al.  The Detection of Gaze Direction: A Stare-In-The-Crowd Effect , 1995, Perception.

[21]  D. Ballard,et al.  Memory Representations in Natural Tasks , 1995, Journal of Cognitive Neuroscience.

[22]  D. Altman,et al.  Multiple significance tests: the Bonferroni method , 1995, BMJ.

[23]  D. S. Wooding,et al.  The relationship between the locations of spatial features and those of fixations made during visual examination of briefly presented images. , 1996, Spatial vision.

[24]  D. Muir,et al.  A demonstration of gaze following in 3- to 6-month-olds , 1997 .

[25]  Hiromitsu Kobayashi,et al.  Unique morphology of the human eye , 1997, Nature.

[26]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[27]  S. Baron-Cohen,et al.  Is There a "Language of the Eyes"? Evidence from Normal Adults, and Adults with Autism or Asperger Syndrome , 1997 .

[28]  A. Kingstone,et al.  The eyes have it! Reflexive orienting is triggered by nonpredictive gaze , 1998 .

[29]  David M. Grayson,et al.  You looking at me? , 1999, Nature.

[30]  S. Baron-Cohen,et al.  Gaze Perception Triggers Reflexive Visuospatial Orienting , 1999 .

[31]  G. Hauske,et al.  Object and scene analysis by saccadic eye-movements: an investigation with higher-order statistics. , 2000, Spatial vision.

[32]  N. Emery,et al.  The eyes have it: the neuroethology, function and evolution of social gaze , 2000, Neuroscience & Biobehavioral Reviews.

[33]  P. Vuilleumier,et al.  Faces call for attention: evidence from patients with visual extinction , 2000, Neuropsychologia.

[34]  J. Haxby,et al.  Distinct representations of eye gaze and identity in the distributed human neural system for face perception , 2000, Nature Neuroscience.

[35]  S. Langton,et al.  The Mutual Influence of Gaze and Head Orientation in the Analysis of Social Attention Direction , 2000, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[36]  H. Wilson,et al.  Perception of head orientation , 2000, Vision Research.

[37]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[38]  M. Hayhoe,et al.  In what ways do eye movements contribute to everyday activities? , 2001, Vision Research.

[39]  N. Lavie,et al.  Changing Faces: A Detection Advantage in the Flicker Paradigm , 2001, Psychological science.

[40]  J. Stevenson.  The cultural origins of human cognition , 2001 .

[41]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[42]  G. Tiberghien,et al.  Gaze direction determination in schizophrenia , 2002, Schizophrenia Research.

[43]  L. Chelazzi,et al.  My eyes want to look where your eyes are looking: Exploring the tendency to imitate another individual's gaze , 2002, Neuroreport.

[44]  J. Piven,et al.  Visual Scanning of Faces in Autism , 2002, Journal of autism and developmental disorders.

[45]  M. Asada,et al.  A developmental approach accelerates learning of joint attention , 2002, Proceedings 2nd International Conference on Development and Learning. ICDL 2002.

[46]  Rechele Brooks,et al.  Gaze following at 12 and 14 months: Do the eyes matter? , 2002 .

[47]  J. O'Doherty,et al.  Automatic and intentional brain responses during evaluation of trustworthiness of faces , 2002, Nature Neuroscience.

[48]  Kerstin Dautenhahn,et al.  Challenges in Building Robots That Imitate People , 2002 .

[49]  K. Dautenhahn,et al.  Imitation in Animals and Artifacts , 2002 .

[50]  Derrick J. Parkhurst,et al.  Modeling the role of salience in the allocation of overt visual attention , 2002, Vision Research.

[51]  J. Henderson.  Human gaze control during real-world scene perception , 2003, Trends in Cognitive Sciences.

[52]  Mark H. Johnson,et al.  Does gaze perception facilitate overt orienting? , 2003 .

[53]  D. Ballard,et al.  What you see is what you need. , 2003, Journal of vision.

[54]  R. Compton.  The interface between emotion and attention: a review of evidence from psychology and neuroscience. , 2003, Behavioral and cognitive neuroscience reviews.

[55]  Alan Kingstone,et al.  Abrupt onsets and gaze direction cues trigger independent reflexive attentional effects , 2003, Cognition.

[56]  Giulio Sandini,et al.  Developmental robotics: a survey , 2003, Connect. Sci..

[57]  Jian-Gang Wang,et al.  Eye gaze estimation from a single image of one eye , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[58]  E. Gordon,et al.  Face to face: visual scanpath evidence for abnormal processing of facial expressions in social phobia , 2004, Psychiatry Research.

[59]  J. Wolfe,et al.  What attributes guide the deployment of visual attention and how do they do it? , 2004, Nature Reviews Neuroscience.

[60]  S. Langton,et al.  The influence of head contour and nose angle on the perception of eye-gaze direction , 2004, Perception & psychophysics.

[61]  Mayu Nishimura,et al.  What are you looking at? Acuity for triadic eye gaze. , 2004, The Journal of general psychology.

[62]  Michael Tomasello,et al.  12- and 18-month-old infants follow gaze to spaces behind barriers. , 2004, Developmental science.

[63]  G. McCarthy,et al.  When Strangers Pass , 2004, Psychological science.

[64]  Max M. Louwerse,et al.  Focusing Attention with Deictic Gestures and Linguistic Expressions , 2005 .

[65]  G. McCarthy,et al.  Neural basis of eye gaze processing deficits in autism. , 2005, Brain : a journal of neurology.

[66]  G. Kuhn,et al.  Magic and Fixation: Now You Don't See it, Now You Do , 2005, Perception.

[67]  Carrick C. Williams,et al.  Eye movements are functional during face learning , 2005, Memory & cognition.

[68]  Julie E. Boland,et al.  Cultural variation in eye movements during scene perception. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[69]  Asha Iyer,et al.  Components of bottom-up gaze allocation in natural images , 2005, Vision Research.

[70]  Jason A. Droll,et al.  Task demands control acquisition and storage of visual information. , 2005, Journal of experimental psychology. Human perception and performance.

[71]  Chen Yu,et al.  The Role of Embodied Intention in Early Lexical Acquisition , 2005, Cogn. Sci..

[72]  Mark H. Johnson,et al.  Processes of change in brain and cognitive development , 2005, Trends in Cognitive Sciences.

[73]  M. Bindemann,et al.  Faces retain attention , 2005, Psychonomic bulletin & review.

[74]  Doris Y. Tsao,et al.  A Cortical Region Consisting Entirely of Face-Selective Cells , 2006, Science.

[75]  Michael A. Arbib,et al.  Action to Language via the Mirror Neuron System: Attention and the minimal subscene , 2006 .

[76]  Heiko Neumann,et al.  Detection of Head Pose and Gaze Direction for Human-Computer Interaction , 2006, PIT.

[77]  Matthew W. Hoffman,et al.  A probabilistic model of gaze imitation and shared attention , 2006, Neural Networks.

[78]  Dejan Todorović,et al.  Geometrical basis of perception of gaze direction , 2006, Vision Research.

[79]  G. Csibra,et al.  Social learning and social cognition: The case for pedagogy , 2006 .

[80]  Stephen V. Shepherd,et al.  Social status gates social attention in monkeys , 2006, Current Biology.

[81]  N. Kanwisher,et al.  The fusiform face area: a cortical region specialized for the perception of faces , 2006, Philosophical Transactions of the Royal Society B: Biological Sciences.

[82]  Antonio Torralba,et al.  Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. , 2006, Psychological review.

[83]  M. Arbib.  Action to language via the mirror neuron system , 2006 .

[84]  Laurent Itti,et al.  The role of memory in guiding attention during natural vision. , 2006, Journal of vision.

[85]  K. Plunkett,et al.  The use of social and salience cues in early word learning. , 2006, Journal of experimental child psychology.

[86]  Lucas Paletta,et al.  Attention in Cognitive Systems. Theories and Systems from an Interdisciplinary Viewpoint , 2008, Lecture Notes in Computer Science.

[87]  P. Hauf,et al.  Object processing during a joint gaze following task , 2007 .

[88]  D. Muir,et al.  Gaze-following : its development and significance , 2007 .

[89]  Monica S. Castelhano,et al.  I See What You See: Eye Movements in Real-World Scenes Are Affected by Perceived Direction of Gaze , 2008, WAPCV.

[90]  R. Jenkins,et al.  Are you looking at me? Neural correlates of gaze adaptation , 2007, Neuroreport.

[91]  A. Todorov,et al.  Predicting political elections from rapid and unreflective face judgments , 2007, Proceedings of the National Academy of Sciences.

[92]  K. Jellinger.  Processes of Change in Brain and Cognitive Development Attention and Performance XXI , 2007 .

[93]  Mohan M. Trivedi,et al.  Head Pose Estimation for Driver Assistance Systems: A Robust Algorithm and Experimental Evaluation , 2007, 2007 IEEE Intelligent Transportation Systems Conference.

[94]  Benjamin W Tatler,et al.  The central fixation bias in scene viewing: selecting an optimal viewing position independently of motor biases and image feature distributions. , 2007, Journal of vision.

[95]  Anna Papafragou,et al.  Does language guide event perception? Evidence from eye movements , 2008, Cognition.

[96]  S. Martinez-Conde,et al.  Attention and awareness in stage magic: turning tricks into research , 2008, Nature Reviews Neuroscience.

[97]  P. Thier,et al.  How precise is gaze following in humans? , 2008, Vision Research.

[98]  M. Tomasello,et al.  Does the chimpanzee have a theory of mind? 30 years later , 2008, Trends in Cognitive Sciences.

[99]  J. Findlay,et al.  Rapid Detection of Person Information in a Naturalistic Scene , 2008, Perception.

[100]  A. Kingstone,et al.  Social Attention and Real-World Scenes: The Roles of Action, Competition and Social Content , 2008, Quarterly journal of experimental psychology.

[101]  Ben Kenward,et al.  The Microstructure of Infants' Gaze as They View Adult Shifts in Overt Attention , 2008 .

[102]  Andrew Zisserman,et al.  Progressive search space reduction for human pose estimation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[103]  C. Koch,et al.  Task-demands can immediately reverse the effects of sensory-driven saliency in complex visual stimuli. , 2008, Journal of vision.

[104]  S. Reader,et al.  Gaze following in monkeys is modulated by observed facial expressions , 2008, Animal Behaviour.

[105]  Doris Y. Tsao,et al.  Patches with Links: A Unified System for Processing Faces in the Macaque Temporal Lobe , 2008, Science.

[106]  Alexander Todorov,et al.  Evaluating face trustworthiness: a model based approach. , 2008, Social cognitive and affective neuroscience.

[107]  J. Barton,et al.  Fixation and saliency during search of natural scenes: The case of visual agnosia , 2009, Neuropsychologia.

[108]  C. Koch,et al.  Faces and text attract gaze independent of the task: Experimental data and computer model. , 2009, Journal of vision.

[109]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[110]  T. Foulsham,et al.  Saliency and scan patterns in the inspection of real-world scenes: Eye movements during encoding and recognition , 2009 .

[111]  Dave S. Kerby,et al.  The effect of head turn on the perception of gaze , 2009, Vision Research.

[112]  Dejan Todorović,et al.  The Effect of Face Eccentricity on the Perception of Gaze Direction , 2009, Perception.

[113]  A. Klin,et al.  Two-year-olds with autism orient to nonsocial contingencies rather than biological motion , 2009, Nature.

[114]  Marianne A. DeAngelus,et al.  Top-down control of eye movements: Yarbus revisited , 2009 .

[115]  M. Doherty,et al.  The rapid development of explicit gaze judgment ability at 3 years. , 2009, Journal of experimental child psychology.

[116]  Michael C. Frank,et al.  Eye-movements reveal attention to social information in autism spectrum disorder , 2009, Neuropsychologia.

[117]  Frédo Durand,et al.  Learning to predict where humans look , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[118]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[119]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[120]  A. Little,et al.  Facial cues of dominance modulate the short-term gaze-cuing effect in human observers , 2009, Proceedings of the Royal Society B: Biological Sciences.

[121]  Matthew H Tong,et al.  SUN: Top-down saliency using natural statistics , 2009, Visual cognition.

[122]  Krista A. Ehinger,et al.  Modelling search for people in 900 scenes: A combined source model of eye guidance , 2009 .

[123]  A. Kingstone,et al.  Saliency does not account for fixations to eyes within social scenes , 2009, Vision Research.

[124]  Jan Zwickel,et al.  How the presence of persons biases eye movements , 2010, Psychonomic bulletin & review.

[125]  Pietro Perona,et al.  Measuring and Predicting Object Importance , 2011, International Journal of Computer Vision.

[126]  Gernot A. Fink,et al.  Saliency-based identification and recognition of pointed-at objects , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[127]  Tom Foulsham,et al.  Gaze allocation in a dynamic situation: Effects of social status and speaking , 2010, Cognition.

[128]  G. Underwood,et al.  The potency of people in pictures: evidence from sequences of eye fixations. , 2010, Journal of vision.

[129]  John J. B. Allen,et al.  Neurophysiological evidence for the influence of past experience on figure-ground perception. , 2010, Journal of vision.

[130]  John M. Henderson,et al.  Attentional selection in scene viewing is object-based , 2010 .

[131]  Rajesh P. N. Rao,et al.  "Social" robots are psychological agents for infants: A test of gaze following , 2010, Neural Networks.

[132]  M. Carrasco.  Visual attention: The past 25 years , 2011, Vision Research.

[133]  Alexander C. Schütz,et al.  Eye movements and perception: a selective review. , 2011, Journal of vision.

[134]  Dicky N. Sihite,et al.  Quantifying the relative influence of photographer bias and viewing strategy on scene viewing , 2011 .

[135]  David H Brainard,et al.  Surface color perception and equivalent illumination models. , 2011, Journal of vision.

[136]  J. Tipples,et al.  Increased gaze following for fearful faces. It depends on what you’re looking for! , 2011, Psychonomic bulletin & review.

[137]  D. Ballard,et al.  Eye guidance in natural vision: reinterpreting salience. , 2011, Journal of vision.

[138]  Hsueh-Cheng Wang,et al.  The Attraction of Visual Attention to Texts in Real-World Scenes: Are Chinese Texts Attractive to Non-Chinese Speakers? , 2011, CogSci.

[139]  L. Itti,et al.  High-throughput classification of clinical populations from natural viewing eye movements , 2012, Journal of Neurology.

[140]  Ali Borji,et al.  Salient Object Detection: A Benchmark , 2012, ECCV.

[141]  Ali Borji,et al.  Boosting bottom-up and top-down visual features for saliency estimation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[142]  Ali Borji,et al.  Exploiting local and global patch rarities for saliency detection , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[143]  Matthias Scheutz,et al.  Adaptive eye gaze patterns in interactions with human and artificial agents , 2012, TIIS.

[144]  L. Itti,et al.  Top-down influences on visual attention during listening are modulated by observer sex , 2012, Vision Research.

[145]  Víctor Leborán,et al.  On the relationship between optical variability, visual saliency, and eye fixations: a computational approach. , 2012, Journal of vision.

[146]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[147]  Nicu Sebe,et al.  Combining Head Pose and Eye Location Information for Gaze Estimation , 2012, IEEE Transactions on Image Processing.

[148]  Yaser Sheikh,et al.  3D Social Saliency from Head-mounted Cameras , 2012, NIPS.

[149]  Gustav Kuhn,et al.  Visual cognition during real social interaction , 2012, Front. Hum. Neurosci..

[150]  J. Kulikowski,et al.  Phases of daylight and the stability of color perception in the near peripheral human retina. , 2012, Journal of vision.

[151]  Albert Ali Salah,et al.  Joint Attention by Gaze Interpolation and Saliency , 2013, IEEE Transactions on Cybernetics.

[152]  Yifan Peng,et al.  Exploring the role of gaze behavior and object detection in scene understanding , 2013, Front. Psychol..

[153]  Jean-Marc Odobez,et al.  Person independent 3D gaze estimation from remote RGB-D cameras , 2013, 2013 IEEE International Conference on Image Processing.

[154]  T. Kanda,et al.  The power of human gaze on infant learning , 2013, Cognition.

[155]  Benjamin W Tatler,et al.  Do as eye say: gaze cueing and language in a real-world social interaction. , 2013, Journal of vision.

[156]  Ali Borji,et al.  Objects do not predict fixations better than early saliency: a re-analysis of Einhauser et al.'s data. , 2013, Journal of vision.

[157]  Kostas Karpouzis,et al.  Visual Focus of Attention in Non-calibrated Environments using Gaze Estimation , 2014, International Journal of Computer Vision.

[158]  Eileen Kowler,et al.  Eye movements while viewing narrated, captioned, and silent videos. , 2013, Journal of vision.

[159]  Ali Borji,et al.  State-of-the-Art in Visual Attention Modeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[160]  Ali Borji,et al.  What stands out in a scene? A study of human explicit saliency judgment , 2013, Vision Research.

[161]  Ali Borji,et al.  Quantitative Analysis of Human-Model Agreement in Visual Saliency Modeling: A Comparative Study , 2013, IEEE Transactions on Image Processing.

[162]  David R. Bull,et al.  Projective image restoration using sparsity regularization , 2013, 2013 IEEE International Conference on Image Processing.

[163]  Andrew Zisserman,et al.  Detecting People Looking at Each Other in Videos , 2014, International Journal of Computer Vision.

[164]  L. Itti,et al.  Defending Yarbus: eye movements reveal observers' task. , 2014, Journal of vision.

[165]  S. Dakin,et al.  Metamorphopsia and letter recognition. , 2014, Journal of vision.

[166]  N. Sebe,et al.  Emotion modulates eye movement patterns and subsequent memory for the gist and details of movie scenes. , 2014, Journal of vision.

[167]  Ali Borji,et al.  What/Where to Look Next? Modeling Top-Down Visual Attention in Complex Interactive Environments , 2014, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[168]  Andreas Bulling,et al.  EyeTab: model-based gaze estimation on unmodified tablet computers , 2014, ETRA.

[169]  Understanding the Link Between Joint Attention and Language , 2014 .

[170]  Ali Borji,et al.  Salient Object Detection: A Benchmark , 2015, IEEE Transactions on Image Processing.