Visual exploration of omnidirectional panoramic scenes

How do we explore the visual environment around us, and how are head and eye movements coordinated during our exploration? To investigate this question, we had observers look at omnidirectional panoramic scenes, composed of both landscape and fractal images, using a virtual reality viewer while their eye and head movements were tracked. We analyzed the spatial distribution of eye fixations and the distribution of saccade directions and the spatial distribution of head positions and the distribution of head shifts, as well as the relation between eye and head movements. The results show that, for landscape scenes, eye and head behavior best fit the allocentric frame defined by the scene horizon, especially when head tilt (i.e., head rotation around the view axis) is considered. For fractal scenes, which have an isotropic texture, eye and head movements were executed primarily along the cardinal directions in world coordinates. The results also show that eye and head movements are closely linked in space and time in a complementary way, with stimulus-driven eye movements predominantly leading the head movements. Our study is the first to systematically examine eye and head movements in a panoramic virtual reality environment, and the results demonstrate that a virtual reality environment constitutes a powerful and informative research alternative to traditional methods for investigating looking behavior.

[1]  Tom Foulsham,et al.  Asymmetries in the direction of saccades during perception of scenes and fractals: Effects of image type and image features , 2010, Vision Research.

[2]  Antonio Torralba,et al.  Modeling global scene factors in attention. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[3]  Tom Foulsham,et al.  Top-down and bottom-up aspects of active search in a real-world environment. , 2014, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[4]  Wolfgang Einhäuser,et al.  Mind the step: complementary effects of an implicit task on eye and head movements in real-life gaze allocation , 2012, Experimental Brain Research.

[5]  Walter F. Bischof,et al.  Turning the (virtual) world around: Patterns in saccade direction vary with picture orientation and shape in virtual reality , 2020, Journal of Vision.

[6]  Benjamin W. Tatler,et al.  Systematic tendencies in scene viewing , 2008 .

[7]  Roland J. Baddeley,et al.  The nature of the visual representations involved in eye movements when walking down the street , 2009 .

[8]  Sascha Weber,et al.  Gaze3DFix: Detecting 3D fixations with an ellipsoidal bounding volume , 2018, Behavior research methods.

[9]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[10]  J. Findlay,et al.  Using the eye–movement system to control the head , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[11]  M. Land,et al.  The Roles of Vision and Eye Movements in the Control of Activities of Daily Living , 1998, Perception.

[12]  Joseph H. Goldberg,et al.  Identifying fixations and saccades in eye-tracking protocols , 2000, ETRA.

[13]  M. Potter,et al.  Pictorial and conceptual representation of glimpsed pictures. , 2004, Journal of experimental psychology. Human perception and performance.

[14]  M. Hayhoe,et al.  In what ways do eye movements contribute to everyday activities? , 2001, Vision Research.

[15]  B. Velichkovsky,et al.  Time course of information processing during scene perception: The relationship between saccade amplitude and fixation duration , 2005 .

[16]  B. Tatler,et al.  Looking and Acting: Vision and eye movements in natural behaviour , 2009 .

[17]  P. Quinn,et al.  Abnormality in face scanning by children with autism spectrum disorder is limited to the eye region: evidence from multi-method analyses of eye tracking data. , 2013, Journal of vision.

[18]  J. Fuller,et al.  Head movement propensity , 2004, Experimental Brain Research.

[19]  Ignace T. C. Hooge,et al.  Is the eye-movement field confused about fixations and saccades? A survey among 124 researchers , 2018, Royal Society Open Science.

[20]  Giulio Sandini,et al.  Control Strategies in the Eye-Head Coordination System , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[21]  D. Straumann,et al.  Torsional vestibulo-ocular reflex during whole-body oscillation in the upright and the supine position. , 2000, Experimental Brain Research.

[22]  E. Freedman.  Coordination of the eyes and head during visual orienting , 2008, Experimental Brain Research.

[23]  Tom Foulsham,et al.  Gaze allocation in a dynamic situation: Effects of social status and speaking , 2010, Cognition.

[24]  Nicola C. Anderson,et al.  Curious eyes: Individual differences in personality predict eye movement behavior in scene-viewing , 2012, Cognition.

[25]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[26]  R. C. Langford.  How People Look at Pictures, A Study of the Psychology of Perception in Art. , 1936 .

[27]  Alan Kingstone,et al.  Balancing energetic and cognitive resources: Memory use during search depends on the orienting effector , 2014, Cognition.

[28]  Krista A. Ehinger,et al.  Recognizing scene viewpoint using panoramic place representation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  P. König,et al.  Gaze allocation in natural stimuli: Comparing free exploration to head-fixed viewing conditions , 2009 .

[30]  Yu Fang,et al.  Eye-Head Coordination for Visual Cognitive Processing , 2015, PloS one.

[31]  T. Foulsham,et al.  The where, what and when of gaze allocation in the lab and the natural environment , 2011, Vision Research.

[32]  N. Mackworth,et al.  The gaze selects informative details within pictures , 1967 .

[33]  Clifton M. Schor,et al.  Neural Control of Eye Movements , 2010 .

[34]  Oleg V. Komogortsev,et al.  Standardization of Automated Analyses of Oculomotor Fixation and Saccadic Behaviors , 2010, IEEE Transactions on Biomedical Engineering.

[35]  Gordon Wetzstein,et al.  Saliency in VR: How Do People Explore Virtual Environments? , 2016, IEEE Transactions on Visualization and Computer Graphics.

[36]  P. Blignaut.  Fixation identification: The optimum threshold for a dispersion algorithm , 2009, Attention, perception & psychophysics.

[37]  L. F. Dell'Osso,et al.  Mental Lives , 1993, Neurology.

[38]  Mary Hayhoe,et al.  Real-time recording and classification of eye movements in an immersive virtual environment. , 2013, Journal of vision.

[39]  D. Ballard,et al.  Eye guidance in natural vision: reinterpreting salience. , 2011, Journal of vision.

[40]  J. Fuller,et al.  Comparison of Head Movement Strategies among Mammals , 1992 .

[41]  T. Foulsham,et al.  What can saliency models predict about eye movements? Spatial and sequential aspects of fixations during encoding and recognition. , 2008, Journal of vision.

[42]  Tom Foulsham,et al.  Eye and head movements are complementary in visual selection , 2017, Royal Society Open Science.

[43]  G. Barnes.  Vestibulo‐ocular function during co‐ordinated head and eye movements to acquire visual targets. , 1979, The Journal of physiology.

[44]  A. Kingstone,et al.  Cognitive Ethology: a new approach for studying human cognition. , 2008, British journal of psychology.

[45]  T. Foulsham,et al.  Saliency and scan patterns in the inspection of real-world scenes: Eye movements during encoding and recognition , 2009 .

[46]  A. Kingstone,et al.  Gaze selection in complex social scenes , 2008 .

[47]  T. Foulsham,et al.  Quarterly Journal of Experimental Psychology: in press Visual saliency and semantic incongruency influence eye movements when , 2022 .

[48]  A. Kingstone,et al.  Saliency does not account for fixations to eyes within social scenes , 2009, Vision Research.

[49]  Derrick J. Parkhurst,et al.  Modeling the role of salience in the allocation of overt visual attention , 2002, Vision Research.

[50]  Eileen Kowler,et al.  Anticipatory smooth eye movements with random-dot kinematograms. , 2012, Journal of vision.

[51]  M. Trivedi,et al.  Head and eye gaze dynamics during visual attention shifts in complex environments. , 2012, Journal of vision.

[52]  H. Goltz,et al.  Static ocular counterroll reflex in skew deviation , 2011, Neurology.

[53]  D Purves,et al.  The distribution of oriented contours in the real world. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[54]  E. Bizzi,et al.  Eye-Head Coordination in Monkeys: Evidence for Centrally Patterned Organization , 1971, Science.

[55]  D. Ballard,et al.  Eye movements in natural behavior , 2005, Trends in Cognitive Sciences.

[56]  B. Heller.  Circular Statistics in Biology, Edward Batschelet. Academic Press, London & New York (1981), 371, Price $69.50 , 1983 .

[57]  Michael C. Dorris,et al.  Combined eye-head gaze shifts to visual and auditory targets in humans , 1996, Experimental Brain Research.

[58]  Tom Foulsham,et al.  Turning the world around: Patterns in saccade direction vary with picture orientation , 2008, Vision Research.

[59]  M. Hayhoe,et al.  The coordination of eye, head, and hand movements in a natural task , 2001, Experimental Brain Research.

[60]  J. Stahl,et al.  Eye-head coordination and the variation of eye-movement accuracy with orbital eccentricity , 2001, Experimental Brain Research.

[61]  T. Foulsham,et al.  Are Fixations in Static Natural Scenes a Useful Predictor of Attention in the Real World? , 2017, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[62]  Peter König,et al.  Human eye-head co-ordination in natural exploration , 2007, Network.

[63]  D. Robinson,et al.  The vestibulo‐ocular reflex during human saccadic eye movements. , 1986, The Journal of physiology.

[64]  Benjamin W Tatler,et al.  The central fixation bias in scene viewing: selecting an optimal viewing position independently of motor biases and image feature distributions. , 2007, Journal of vision.

[65]  Alain Berthoz,et al.  Head Kinematics during Complex Movements , 1992 .

[66]  Li Li,et al.  The Accuracy and Precision of Position and Orientation Tracking in the HTC Vive Virtual Reality System for Scientific Research , 2017, i-Perception.