Spatial Sound and Multimodal Interaction in Immersive Environments

Spatial sound and interactivity are key elements of investigation at the Sound And Music Computing master program at Aalborg University Copenhagen. We present a collection of research directions and recent results from work in these areas, with the focus on our multifaceted approaches to two primary problem areas: 1) creation of interactive spatial audio experiences for immersive virtual and augmented reality scenarios, and 2) production and mixing of spatial audio for cinema, music, and other artistic contexts. Several ongoing research projects are described, wherein the latest developments are discussed. These include elements in which we have provided sonic interaction in virtual environments, interactivity with volumetric sound sources using VBAP and Wave Field Synthesis (WFS), and binaural sound for virtual environments and spatial audio mixing. We show that the variety of approaches presented here are necessary in order to optimize interactivity with spatial audio for each particular type of task.

[1]  Cumhur Erkut,et al.  Parametric time-frequency representation of spatial sound in virtual worlds , 2012, TAP.

[2]  F. Melchior,et al.  Wave Field Syntheses in Combination with 2D Video Projection , 2003 .

[3]  Mary C. Whitton,et al.  Improved Redirection with Distractors: A large-scale-real-walking locomotion interface and its effect on navigation in virtual environments , 2010, 2010 IEEE Virtual Reality Conference (VR).

[4]  Dan Overholt,et al.  Towards an Interface for Music Mixing based on Smart Tangibles and Multitouch , 2013, NIME.

[5]  Mary C. Whitton,et al.  Evaluation of Reorientation Techniques and Distractors for Walking in Large Virtual Environments , 2009, IEEE Transactions on Visualization and Computer Graphics.

[6]  Frank Melchior,et al.  Combining Wave Field Synthesis and Multi-Viewer Stereo Displays , 2006, IEEE Virtual Reality Conference (VR 2006).

[7]  Li-Wei Chan,et al.  Real-Time 3D Model-Based Gesture Tracking for Multimedia Control , 2010, 2010 20th International Conference on Pattern Recognition.

[8]  Luiz Velho,et al.  Kinect and RGBD Images: Challenges and Applications , 2012, 2012 25th SIBGRAPI Conference on Graphics, Patterns and Images Tutorials.

[9]  Luc Van Gool,et al.  Real-time 3D hand gesture interaction with a robot for understanding directions from humans , 2011, 2011 RO-MAN.

[10]  Frank Melchior,et al.  On the Use of a Haptic Feedback Device for Sound Source Control in Spatial Audio Systems , 2013 .

[11]  Jung-Woo Choi.  Extension of perceived source width using sound field reproduction systems , 2013 .

[12]  Frank Melchior,et al.  Authoring and user interaction for the production of wave field synthesis content in an augmented reality system , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[13]  Ville Pulkki,et al.  Virtual Sound Source Positioning Using Vector Base Amplitude Panning , 1997 .

[14]  Steven Gelineck,et al.  Towards a more flexible and creative music mixing interface , 2013, CHI Extended Abstracts.

[15]  Tom Mens,et al.  Statechart modelling of interactive gesture-based applications , 2011 .

[16]  Cumhur Erkut,et al.  Efficient Spatial Sound Synthesis for Virtual Worlds , 2009 .

[17]  Meredith Ringel Morris,et al.  Kinected browser: depth camera interaction for the web , 2012, ITS '12.

[18]  Ville Pulkki,et al.  Synthesis of Spatially Extended Virtual Sources with Time-Frequency Decomposition of Mono Signals , 2014 .

[19]  Sascha Spors,et al.  Two Physical Models for Spatially Extended Virtual Sound Sources , 2011 .

[20]  Emmanouil Potetsianakis,et al.  A Kinect-based framework for better user experience in real-time audiovisual content manipulation , 2014, 2014 International Conference on Telecommunications and Multimedia (TEMU).

[21]  Steven Gelineck,et al.  An Exploratory Evaluation of User Interfaces for 3D Audio Mixing , 2015 .

[22]  Daniel Västfjäll,et al.  Better Presence and Performance in Virtual Environments by Improved Binaural Sound Rendering , 2002 .

[23]  Stefania Serafin,et al.  Audio-visual attractors for capturing attention to the screens when walking in CAVE systems , 2014, 2014 IEEE Virtual Reality (VR).

[24]  C. Avendano,et al.  The CIPIC HRTF database , 2001, Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575).

[25]  Peter Driessen,et al.  Sensor fusion: Towards a fully expressive 3D music control interface , 2011, Proceedings of 2011 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing.

[26]  Tapio Lokki,et al.  Localizing Sound Sources in a CAVE-Like Virtual Environment with Loudspeaker Array Reproduction , 2007, PRESENCE: Teleoperators and Virtual Environments.

[27]  F. Grani,et al.  Audio-visual attractors for capturing attention to the screens when walking in CAVE systems , 2014, 2014 IEEE VR Workshop: Sonic Interaction in Virtual Environments (SIVE).

[28]  Neil M. Robertson,et al.  A proposed gesture set for the control of industrial collaborative robots , 2012, 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.

[29]  Stéphane Huot,et al.  A Morphological Analysis of Audio-Objects and their Control Methods for 3D Audio , 2014, NIME.

[30]  Durand R. Begault,et al.  3-D Sound for Virtual Reality and Multimedia Cambridge , 1994 .

[31]  Stefania Serafin,et al.  Design and evaluation of Binaural auditory rendering for CAVEs , 2014, 2014 IEEE Virtual Reality (VR).

[32]  Kenton O'Hara,et al.  Exploring the potential for touchless interaction in image-guided interventional radiology , 2011, CHI.

[33]  Jean-Marc Jot,et al.  Real-time spatial processing of sounds for music, multimedia and interactive human-computer interfaces , 1999, Multimedia Systems.

[34]  R. Rabenstein,et al.  PHYSICAL MODELING FOR SPATIAL SOUND SYNTHESIS , 2009 .

[35]  Wolfgang Fohl,et al.  A Gesture Control Interface for a Wave Field Synthesis System , 2013, NIME.

[36]  Jörg Müller,et al.  The boomRoom: mid-air direct interaction with virtual sound sources , 2014, CHI.

[37]  Dan Xu,et al.  Real-time dynamic gesture recognition system based on depth perception for robot navigation , 2012, 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[38]  Chris Pike,et al.  Musical Movements—Gesture Based Audio Interfaces , 2011 .

[39]  N. Vidakis,et al.  Multimodal natural user interaction for multiple applications: The gesture — Voice example , 2012, 2012 International Conference on Telecommunications and Multimedia (TEMU).