Fundamentals of Agent Perception and Attention Modelling

Perception and attention mechanisms are of great importance for entities situated within complex dynamic environments. With roles extending greatly beyond passive information services about the external environment, such mechanisms actively prioritise, augment and expedite information to ensure that the potentially relevant is made available so appropriate action can take place. Here, we describe the rationale behind endowing artificial entities, or virtual agents, with real-time perception and attention systems. We cover the fundamentals of designing and building such systems. Once equipped, the resulting agents can achieve a more substantial connection with their environment for the purposes of reacting, planning, decision making and, ultimately, behaving.

[1]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[2]  M. Posner,et al.  Orienting of Attention* , 1980, The Quarterly journal of experimental psychology.

[3]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[4]  C. Eriksen,et al.  Allocation of attention in the visual field. , 1985, Journal of experimental psychology. Human perception and performance.

[5]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[6]  Alex Waibel,et al.  Readings in speech recognition , 1990 .

[7]  Proceedings of the 21st annual conference on Computer graphics and interactive techniques , 1994 .

[8]  Demetri Terzopoulos,et al.  Artificial fishes: physics, locomotion, perception, behavior , 1994, SIGGRAPH.

[9]  Daniel Thalmann,et al.  Navigation for digital actors based on synthetic vision, memory, and learning , 1995, Comput. Graph..

[10]  Demetri Terzopoulos,et al.  Animat vision: Active vision in artificial animals , 1995, Proceedings of IEEE International Conference on Computer Vision.

[11]  Kristinn R. Thórisson.  Gandalf: an embodied humanoid capable of real-time multimodal dialogue with people , 1997, AGENTS '97.

[12]  Bruce Blumberg.  Go with the flow: synthetic vision for autonomous animated creatures , 1997, AGENTS '97.

[13]  Gary Bradski,et al.  Computer Vision Face Tracking For Use in a Perceptual User Interface , 1998 .

[14]  Jean-Claude Latombe,et al.  Fast synthetic vision, memory, and learning models for virtual humans , 1999, Proceedings Computer Animation 1999.

[15]  Daniel Thalmann,et al.  An Efficient and Flexible Perception Pipeline for Autonomous Agents , 1999, Comput. Graph. Forum.

[16]  Norman I. Badler,et al.  Where to Look? Automating Attending Behaviors of Virtual Human Characters , 1999, Agents.

[17]  S. Drucker,et al.  The Role of Eye Gaze in Avatar Mediated Conversational Interfaces , 2000 .

[18]  Claudio M. Privitera,et al.  Algorithms for Defining Visual Regions-of-Interest: Comparison with Eye Fixations , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Catherine Pelachaud,et al.  Eye Communication in a Conversational 3D Synthetic Agent , 2000, AI Commun..

[20]  Craig W. Reynolds.  Interaction with Groups of Autonomous Characters , 2000 .

[21]  M. F. P. Gillies,et al.  Practical behavioural animation based on vision and attention , 2001 .

[22]  Bruce Blumberg,et al.  A Layered Brain Architecture for Synthetic Creatures , 2001, IJCAI.

[23]  Taku Komura,et al.  Topology matching for fully automatic similarity estimation of 3D shapes , 2001, SIGGRAPH.

[24]  Mel Slater,et al.  The impact of eye gaze on communication using humanoid avatars , 2001, CHI.

[25]  Carlos Delgado-Mata,et al.  Communicating Emotion in Virtual Environments through Artificial Scents , 2001, IVA.

[26]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[27]  K. Scherer,et al.  Appraisal processes in emotion: Theory, methods, research. , 2001 .

[28]  Dirk Heylen,et al.  Making agents gaze naturally - does it work? , 2002, AVI '02.

[29]  Roel Vertegaal,et al.  Explaining effects of eye gaze on mediated group conversations:: amount or synchronization? , 2002, CSCW '02.

[30]  R. Hill.  Perceptual Attention in Virtual Humans: Towards Realistic and Believable Gaze Behaviors , 2002 .

[31]  Carol O'Sullivan,et al.  Synthetic Vision and Memory for Autonomous Virtual Humans , 2002, Comput. Graph. Forum.

[32]  Norman I. Badler,et al.  Eyes alive , 2002, ACM Trans. Graph..

[33]  Laurent Itti,et al.  Realistic avatar eye and head animation using a neurobiological model of visual attention , 2004, SPIE Optics + Photonics.

[34]  Christopher E. Peters,et al.  Bottom-up visual attention for virtual human animation , 2003, Proceedings 11th IEEE International Workshop on Program Comprehension.

[35]  Themis Panayiotopoulos,et al.  Programmable Agent Perception in Intelligent Virtual Environments , 2003, IVA.

[36]  Antonio Camurri,et al.  Analysis of Expressive Gesture: The EyesWeb Expressive Gesture Processing Library , 2003, Gesture Workshop.

[37]  Antonio Camurri,et al.  Gesture-Based Communication in Human-Computer Interaction , 2003, Lecture Notes in Computer Science.

[38]  Nicolas Courty,et al.  Visual perception based on salient features , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[39]  Angélica de Antonio Jiménez,et al.  Introducing human-like hearing perception in intelligent virtual agents , 2003, AAMAS '03.

[40]  Michael Wooldridge,et al.  Proceedings of the second international joint conference on autonomous agents and multiagent systems (AAMAS) , 2003 .

[41]  Heinz Hügli,et al.  Empirical Validation of the Saliency-based Model of Visual Attention , 2003 .

[42]  António Lucas Soares,et al.  An Efficient Synthetic Vision System for 3D Multi-character Systems , 2003, IVA.

[43]  Ivan Laptev,et al.  On Space-Time Interest Points , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[44]  Trevor Darrell,et al.  From conversational tooltips to grounded discourse: head poseTracking in interactive dialog systems , 2004, ICMI '04.

[45]  Maja Pantic,et al.  Particle filtering with factorized likelihoods for tracking facial features , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[46]  Elisabeth André,et al.  Where Do They Look? Gaze Behaviors of Multiple Users Interacting with an Embodied Conversational Agent , 2005, IVA.

[47]  I. Patras,et al.  Spatiotemporal salient points for visual recognition of human actions , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[48]  Randall W. Hill,et al.  Modeling Dynamic Perceptual Attention in Complex Virtual Environments , 2005, IVA.

[49]  Witold Pedrycz,et al.  Face recognition: A study in information fusion using fuzzy integral , 2005, Pattern Recognit. Lett..

[50]  Yannis Avrithis,et al.  Handling Uncertainty in Video Analysis with Spatiotemporal Visual Attention , 2005, The 14th IEEE International Conference on Fuzzy Systems, 2005. FUZZ '05..

[51]  Gérard Bailly,et al.  Proceedings of the 2005 joint conference on Smart objects and ambient intelligence - innovative context-aware services: usages and technologies, sOc-EUSAI '05, Grenoble, France, October 12-14, 2005 , 2005, sOc-EUSAI.

[52]  Stephan Raidt,et al.  Basic components of a face-to-face interaction with a conversational agent: mutual attention and deixis , 2005, sOc-EUSAI '05.

[53]  Ipke Wachsmuth,et al.  Deixis: How to Determine Demonstrated Objects Using a Pointing Cone , 2005, Gesture Workshop.

[54]  Daniel Thalmann,et al.  Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/cav.148 , 2022 .

[55]  Christopher E. Peters.  Evaluating Perception of Interaction Initiation in Virtual Environments Using Humanoid Agents , 2006, ECAI.

[56]  Nicolas Courty,et al.  Gesture in Human-Computer Interaction and Simulation , 2006 .

[57]  Yannis Avrithis,et al.  Bottom-up spatiotemporal visual attention model for video analysis , 2007 .

[58]  Antonio Camurri,et al.  Developing multimodal interactive systems with EyesWeb XMI , 2007, NIME '07.

[59]  Norman I. Badler,et al.  Generating Sequence of Eye Fixations Using Decision-Theoretic Attention Model , 2007, WAPCV.

[60]  Ginevra Castellano,et al.  Recognising Human Emotions from Body Movement and Gesture Dynamics , 2007, ACII.

[61]  Gérard Bailly,et al.  Scrutinizing Natural Scenes: Controlling the Gaze of an Embodied Conversational Agent , 2007, IVA.

[62]  M. Bartlett,et al.  Machine Analysis of Facial Expressions , 2007 .

[63]  K. Scherer,et al.  Automated Analysis of Body Movement in Emotionally Expressive Piano Performances , 2008 .

[64]  Laurent Itti,et al.  Applying computational tools to predict gaze direction in interactive visual environments , 2008, TAP.

[65]  Gary R. Bradski,et al.  Learning OpenCV - computer vision with the OpenCV library: software that sees , 2008 .

[66]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[67]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..