Gazing at Social Interactions Between Foraging and Decision Theory

Finding the underlying principles of social attention in humans seems to be essential for the design of the interaction between natural and artificial agents. Here, we focus on the computational modeling of gaze dynamics as exhibited by humans when perceiving socially relevant multimodal information. The audio-visual landscape of social interactions is distilled into a number of multimodal patches that convey different social value, and we work under the general frame of foraging as a tradeoff between local patch exploitation and landscape exploration. We show that the spatio-temporal dynamics of gaze shifts can be parsimoniously described by Langevin-type stochastic differential equations triggering a decision equation over time. In particular, value-based patch choice and handling is reduced to a simple multi-alternative perceptual decision making that relies on a race-to-threshold between independent continuous-time perceptual evidence integrators, each integrator being associated with a patch.

[1]  Michael L. Platt,et al.  Spontaneous social orienting and gaze following in ringtailed lemurs (Lemur catta) , 2007, Animal Cognition.

[2]  Piotr Napieralski,et al.  A model of saliency-based visual attention for movie retrospection , 2017, 2017 18th International Symposium on Electromagnetic Fields in Mechatronics, Electrical and Electronic Engineering (ISEF) Book of Abstracts.

[3]  Otto Lappi,et al.  A new and general approach to signal denoising and eye movement classification based on segmented linear regression , 2017, Scientific Reports.

[4]  Yukie Nagai.  Stability and sensitivity of bottom-up visual attention for dynamic scene analysis , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[5]  Dana H. Ballard,et al.  Animate Vision , 1991, Artif. Intell..

[6]  H S Pennypacker,et al.  On Behavioral Analysis , 1981, The Behavior analyst.

[7]  M. Gamer,et al.  Social content and emotional valence modulate gaze fixations in dynamic scenes , 2018, Scientific Reports.

[8]  Y. Nagai.  From bottom-Up visual attention to robot action learning , 2009, 2009 IEEE 8th International Conference on Development and Learning.

[9]  Marcus Nyström,et al.  Eye tracker data quality: what it is and how to measure it , 2012, ETRA.

[10]  G. Metta,et al.  Robots As Intentional Agents: Using Neuroscientific Methods to Make Robots Appear More Social , 2017, Front. Psychol..

[11]  Ali Borji,et al.  Saliency Prediction in the Deep Learning Era: Successes and Limitations , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  M. Land.  Eye movements and the control of actions in everyday life , 2006, Progress in Retinal and Eye Research.

[13]  Brian Scassellati,et al.  Active vision for sociable robots , 2001, IEEE Trans. Syst. Man Cybern. Part A.

[14]  Jorge Dias,et al.  Attentional Mechanisms for Socially Interactive Robots–A Survey , 2014, IEEE Transactions on Autonomous Mental Development.

[15]  Feng He,et al.  Find Who to Look at: Turning From Action to Saliency , 2018, IEEE Transactions on Image Processing.

[16]  B. Scassellati,et al.  Social eye gaze in human-robot interaction , 2017, J. Hum. Robot Interact..

[17]  G. Pezzulo,et al.  Integrating Reinforcement-Learning , Accumulator Models , and Motor-Primitives to Study Action Selection and Reaching in Monkeys , 2006 .

[18]  J. Staab,et al.  The influence of anxiety on ocular motor control and gaze. , 2014, Current opinion in neurology.

[19]  Karl J. Friston,et al.  Scene Construction, Visual Foraging, and Active Inference , 2016, Front. Comput. Neurosci..

[20]  D. Ballard,et al.  Eye guidance in natural vision: reinterpreting salience. , 2011, Journal of vision.

[21]  George K. Matsopoulos,et al.  Discrete states of attention during active visual fixation revealed by Markovian analysis of the time series of intrusive saccades , 2016, Neuroscience.

[22]  Jochen Ditterich,et al.  A Comparison between Mechanisms of Multi-Alternative Perceptual Decision Making: Ability to Explain Human Behavior, Predictions for Neurophysiology, and Relationship with Decision Theory , 2010, Front. Neurosci..

[23]  C. Llano.  On optimal use of a patchy environment: archaeobotany in the Argentinean Andes (Argentina) , 2015 .

[24]  Giulio Sandini,et al.  Motor biases in visual attention for a humanoid robot , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[25]  Ignace T. C. Hooge,et al.  Is the eye-movement field confused about fixations and saccades? A survey among 124 researchers , 2018, Royal Society Open Science.

[26]  Thomas T. Hills.  Animal Foraging and the Evolution of Goal-Directed Cognition , 2006, Cogn. Sci..

[27]  R. Macarthur,et al.  On Optimal Use of a Patchy Environment , 1966, The American Naturalist.

[28]  Pamela Effrein Sandstrom,et al.  Information Foraging Theory: Adaptive Interaction with Information , 2010, J. Assoc. Inf. Sci. Technol..

[29]  C. Klein,et al.  Typical and Atypical Development of Eye Movements , 2019, Eye Movement Research.

[30]  Benjamin W. Tatler,et al.  Systematic tendencies in scene viewing , 2008 .

[31]  Jeremy M Wolfe,et al.  When is it time to move to the next raspberry bush? Foraging rules in human visual search. , 2013, Journal of vision.

[32]  R. Hessels.  How does gaze to faces support face-to-face interaction? A review and perspective , 2020, Psychonomic bulletin & review.

[33]  Krista A. Ehinger,et al.  When is it time to move to the next map? Optimal foraging in guided visual search , 2016, Attention, Perception, & Psychophysics.

[34]  Don S. Lemons,et al.  An Introduction to Stochastic Processes in Physics , 2002 .

[35]  E. Charnov.  Optimal foraging, the marginal value theorem. , 1976, Theoretical population biology.

[36]  Brian Scassellati,et al.  A Behavioral Analysis of Computational Models of Visual Attention , 2007, International Journal of Computer Vision.

[37]  Dinesh Manocha,et al.  DGaze: CNN-Based Gaze Prediction in Dynamic Scenes , 2020, IEEE Transactions on Visualization and Computer Graphics.

[38]  Marcus Nyström,et al.  A vector-based, multidimensional scanpath similarity measure , 2010, ETRA.

[39]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[40]  Scott D. Brown,et al.  Diffusion Decision Model: Current Issues and History , 2016, Trends in Cognitive Sciences.

[41]  Bo Liu,et al.  Human Gaze Assisted Artificial Intelligence: A Review , 2020, IJCAI.

[42]  Marco Gori,et al.  Gravitational Laws of Focus of Attention , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Ahmad Zibafar,et al.  State-of-the-Art Visual Merchandising Using a Fashionable Social Robot: RoMa , 2019 .

[44]  K. Berridge,et al.  Parsing reward , 2003, Trends in Neurosciences.

[45]  Emily Zane,et al.  Perceptions of self and other: Social judgments and gaze patterns to videos of adolescents with and without autism spectrum disorder , 2018, Autism : the international journal of research and practice.

[46]  Jim Euchner,et al.  Spectrum Disorder , 2012 .

[47]  Olivier Le Meur,et al.  Predicting image influence on visual saliency distribution: the focal and ambient dichotomy , 2020, ETRA Short Papers.

[48]  O. Meur,et al.  Introducing context-dependent and spatially-variant viewing biases in saccadic models , 2016, Vision Research.

[49]  Giuseppe Boccignone,et al.  Ecological Sampling of Gaze Shifts , 2014, IEEE Transactions on Cybernetics.

[50]  Tony Charman,et al.  Simulating interaction: Using gaze-contingent eye-tracking to measure the reward value of social signals in toddlers with and without autism , 2017, Developmental Cognitive Neuroscience.

[51]  Giuliano Grossi,et al.  On Gaze Deployment to Audio-Visual Cues of Social Interactions , 2020, IEEE Access.

[52]  Jan Theeuwes,et al.  ScanMatch: A novel method for comparing fixation sequences , 2010, Behavior research methods.

[53]  Gijs A. Holleman,et al.  The ‘Real-World Approach’ and Its Problems: A Critique of the Term Ecological Validity , 2020, Frontiers in Psychology.

[54]  Ran R. Hassin,et al.  A novel perceptual trait: gaze predilection for faces during visual exploration , 2019, Scientific Reports.

[55]  D. Robinson,et al.  Shared neural control of attentional shifts and eye movements , 1996, Nature.

[56]  Yiannis Aloimonos,et al.  Active vision , 2004, International Journal of Computer Vision.

[57]  Francesco Rea,et al.  Speech Envelope Dynamics for Noise-Robust Auditory Scene Analysis in Robotics , 2020 .

[58]  J. McNamara.  Optimal patch use in a stochastic environment , 1982 .

[59]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[60]  S. Martinez-Conde,et al.  An oculomotor continuum from exploration to fixation , 2013, Proceedings of the National Academy of Sciences.

[61]  Jonathan D. Cohen,et al.  The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. , 2006, Psychological review.

[62]  C. Klein,et al.  Social Visual Perception Under the Eye of Bayesian Theories in Autism Spectrum Disorder Using Advanced Modeling of Spatial and Temporal Parameters , 2020, Frontiers in Psychiatry.

[63]  E. Zane,et al.  Perceptions of Self and Other : Social judgments and gaze patterns to videos of adolescents with and without ASD , 2018 .

[64]  Ruzena Bajcsy,et al.  Active and exploratory perception , 1992, CVGIP Image Underst..

[65]  Giuliano Grossi,et al.  Give Ear to My Face: Modelling Multimodal Attention to Social Interactions , 2018, ECCV Workshops.

[66]  Gordon Cheng,et al.  Attention-based active visual search for mobile robots , 2018, Autonomous Robots.

[67]  A. Rangel,et al.  Multialternative drift-diffusion model predicts the relationship between visual fixations and choice in value-based decisions , 2011, Proceedings of the National Academy of Sciences.

[68]  Tom Foulsham,et al.  Scenes, Saliency Maps and Scanpaths , 2019, Eye Movement Research.

[69]  Esa Rahtu,et al.  Deep Audio-Visual Saliency: Baseline Model and Data , 2020, ETRA Short Papers.

[70]  Ali Borji,et al.  State-of-the-Art in Visual Attention Modeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[71]  Edward Vul,et al.  A Bayesian Optimal Foraging Model of Human Visual Search , 2012, Psychological science.

[72]  B. Velichkovsky,et al.  Time course of information processing during scene perception: The relationship between saccade amplitude and fixation duration , 2005 .

[73]  Zhi Liu,et al.  Saccadic model of eye movements for free-viewing condition , 2015, Vision Research.

[74]  Kai Vogeley,et al.  Distinguishing Social From Private Intentions Through the Passive Observation of Gaze Cues , 2019, Front. Hum. Neurosci..

[75]  Antonio Torralba,et al.  Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. , 2006, Psychological review.

[76]  Tom Foulsham,et al.  Gaze allocation in a dynamic situation: Effects of social status and speaking , 2010, Cognition.

[77]  T. Foulsham,et al.  It depends on how you look at it: Scanpath comparison in multiple dimensions with MultiMatch, a vector-based approach , 2012, Behavior Research Methods.

[78]  Evan F. Risko,et al.  Social attention with real versus reel stimuli: toward an empirical approach to concerns about ecological validity , 2012, Front. Hum. Neurosci..

[79]  F. Bartumeus,et al.  Optimal search behavior and classic foraging theory , 2009 .

[80]  W. Einhäuser,et al.  Attention in natural scenes: Affective-motivational factors guide gaze independently of visual salience , 2017, Vision Research.