Sound localization and multi-modal steering for autonomous virtual agents

With the increasing realism of interactive applications, there is a growing need for harnessing additional sensory modalities such as hearing. While the synthesis and propagation of sounds in virtual environments has been explored, there has been little work that addresses sound localization and its integration into behaviors for autonomous virtual agents. This paper develops a framework that enables autonomous virtual agents to localize sounds in dynamic virtual environments, subject to distortion effects due to attenuation, reflection and diffraction from obstacles, as well as interference between multiple audio signals. We additionally integrate hearing into standard predictive collision avoidance techniques and couple it with vision to allow agents to react to what they see and hear, while navigating in virtual environments.

[1]  Daniel Thalmann,et al.  A Sound Propagation Model for Interagents Communication , 2000, Virtual Worlds.

[2]  Petros Faloutsos,et al.  Egocentric affordance fields in pedestrian steering , 2009, I3D '09.

[3]  Hugh F. Durrant-Whyte,et al.  A solution to the simultaneous localization and map building (SLAM) problem , 2001, IEEE Trans. Robotics Autom..

[4]  J. Pettré,et al.  A synthetic-vision based steering approach for crowd simulation , 2010, ACM Trans. Graph..

[5]  H. Gaskell.  The precedence effect , 1983, Hearing Research.

[6]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[7]  Mubbasir Kapadia,et al.  Navigation and steering for autonomous virtual humans. , 2013, Wiley interdisciplinary reviews. Cognitive science.

[8]  Norman I. Badler,et al.  Virtual Crowds: Methods, Simulation, and Control , 2008, Virtual Crowds: Methods, Simulation, and Control.

[9]  Norman I. Badler,et al.  Multi-domain real-time planning in dynamic environments , 2013, SCA '13.

[10]  Thomas A. Funkhouser,et al.  A beam tracing approach to acoustic modeling for interactive virtual environments , 1998, SIGGRAPH.

[11]  Glenn Reinman,et al.  Footstep navigation for dynamic crowds , 2011, SI3D.

[12]  Pawel Strumillo,et al.  Advances in Sound Localization , 2011 .

[13]  Paris Smaragdis,et al.  A Wrapped Kalman Filter for Azimuthal Speaker Tracking , 2013, IEEE Signal Processing Letters.

[14]  Chen Shen,et al.  Synthesizing sounds from rigid-body simulations , 2002, SCA '02.

[15]  R. Klatzky,et al.  Assessing auditory distance perception using perceptually directed action , 1998, Perception & psychophysics.

[16]  Ravish Mehra,et al.  Precomputed wave simulation for real-time sound propagation of dynamic sources in complex scenes , 2010, SIGGRAPH 2010.

[17]  F. Ihlenburg.  Finite Element Analysis of Acoustic Scattering , 1998 .

[18]  Thomas Funkhouser,et al.  A beam tracing method for interactive architectural acoustics. , 2004, The Journal of the Acoustical Society of America.

[19]  Norman I. Badler,et al.  SPREAD: sound propagation and perception for autonomous agents in dynamic environments , 2013, SCA '13.

[20]  Jernej Barbic,et al.  Precomputed acoustic transfer: output-sensitive, accurate sound generation for geometrically complex vibration sources , 2006, ACM Trans. Graph..

[21]  Ming C. Lin,et al.  Precomputed wave simulation for real-time sound propagation of dynamic sources in complex scenes , 2010, ACM Trans. Graph..

[22]  Rahul Narain,et al.  Efficient and accurate sound propagation using adaptive rectangular decomposition. , 2009, IEEE transactions on visualization and computer graphics.

[23]  Stéphane Donikian,et al.  A synthetic-vision based steering approach for crowd simulation , 2010, SIGGRAPH 2010.

[24]  Jie Huang,et al.  Sound localization in reverberant environment based on the model of the precedence effect , 1997 .

[25]  H S Colburn,et al.  The precedence effect. , 1999, The Journal of the Acoustical Society of America.

[26]  Dinesh Manocha,et al.  Generalized velocity obstacles , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[27]  Augusto Sarti,et al.  REAL TIME MODELING OF ACOUSTIC PROPAGATION IN COMPLEX ENVIRONMENTS , 2004 .

[28]  Dinesh Manocha,et al.  Reciprocal Velocity Obstacles for real-time multi-agent navigation , 2008, 2008 IEEE International Conference on Robotics and Automation.

[29]  N. Badler,et al.  7-2014 ADAPT : The Agent Development and Prototyping Testbed , 2016 .

[30]  R. D. Ciskowski,et al.  Boundary element methods in acoustics , 1991 .

[31]  Daniel Thalmann,et al.  Crowd Simulation, Second Edition , 2013 .

[32]  Keith D. Martin,et al.  A computational model of spatial hearing , 1995 .

[33]  Simon N. Chandler-Wilde,et al.  Boundary element methods for acoustics , 2007 .

[34]  Dinesh Manocha,et al.  An efficient GPU-based time domain solver for the acoustic wave equation , 2012 .

[35]  Norman I. Badler,et al.  Controlling individual agents in high-density crowd simulation , 2007, SCA '07.

[36]  Demetri Terzopoulos,et al.  A decision network framework for the behavioral animation of virtual humans , 2007, SCA '07.

[37]  George Drettakis,et al.  Fast modal sounds with scalable frequency-domain synthesis , 2008, ACM Trans. Graph..

[38]  Demetri Terzopoulos,et al.  Autonomous pedestrians , 2005, SCA '05.

[39]  Dinesh Manocha,et al.  The Hybrid Reciprocal Velocity Obstacle , 2011, IEEE Transactions on Robotics.

[40]  Jont B. Allen,et al.  Image method for efficiently simulating small‐room acoustics , 1976 .

[41]  Helbing,et al.  Social force model for pedestrian dynamics. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[42]  Glenn Reinman,et al.  A modular framework for adaptive agent-based steering , 2011, SI3D.

[43]  Dinesh K. Pai,et al.  Precomputed acoustic transfer: output-sensitive, accurate sound generation for geometrically complex vibration sources , 2006, SIGGRAPH 2006.

[44]  Y. Kagawa,et al.  Discrete Huygens' model approach to sound wave propagation , 1998 .

[45]  Sébastien Paris,et al.  Pedestrian Reactive Navigation for Crowd Simulation: a Predictive Approach , 2007, Comput. Graph. Forum.