A Platform for Audiovisual Telepresence Using Model- and Data-Based Wave-Field Synthesis

We present a platform for real-time transmission of immersive audiovisual impressions using modeland databased audio wave-field analysis/synthesis and panoramic video capturing/projection. The audio subsystem considered in this paper is based on microphone arrays with different element counts and directivities as well as weakly directional loudspeaker arrays. We report on both linear and circular setups that feed different wave-field synthesis systems. In an attempt to extend this, we present first findings for a data-based approach derived using experimental simulations. This data-based wave-field analysis/synthesis (WFAS) approach uses a combination of cylindrical-harmonic decomposition of cardioid array signals and enforces causal plane wave synthesis by angular windowing and a directional delay term. Specifically, our contributions include (1) a high-resolution telepresence environment that is omnidirectional in both the auditory and visual modality, as well as (2) a study of data-based WFAS realistic microphone directivities as a contribution towards for real-time holophonic reproduction.

[1]  E. Verheijen.  Sound reproduction by wave field synthesis , 1998 .

[2]  J. Sonke,et al.  Array technology for acoustic wave field analysis in enclosures , 1997 .

[3]  A. Berkhout,et al.  Applied Seismic Wave Theory , 1987 .

[4]  Anup Basu,et al.  Panoramic video with predictive windows for telepresence applications , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[5]  Diemer de Vries,et al.  Extraction of 3D Information from Circular Array Measurements for Auralization with Wave Field Synthesis , 2007, EURASIP J. Adv. Signal Process..

[6]  Darren Brett Ward.  Theory and application of broadband frequency invariant beamforming , 1996 .

[7]  Naokazu Yokoya,et al.  High-resolution panoramic movie generation from video streams acquired by an omnidirectional multi-camera system , 2003, Proceedings of IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI2003..

[8]  Diemer de Vries,et al.  Improved microphone array configurations for auralization of sound fields by Wave Field Synthesis. , 2002 .

[9]  Jan Baan,et al.  A wave field extrapolation approach to acoustical modeling in enclosed spaces , 1999 .

[10]  Ville Pulkki,et al.  Virtual Sound Source Positioning Using Vector Base Amplitude Panning , 1997 .

[11]  Mickael Tanter,et al.  Sound focusing in rooms: the time-reversal approach. , 2003, The Journal of the Acoustical Society of America.

[12]  Michael S. Brandstein,et al.  Real-Time Automated Video and Audio Capture with Multiple Cameras and Microphones , 2001, J. VLSI Signal Process..

[13]  D. de Vries,et al.  Wave field synthesis and analysis using array technology , 1999, Proceedings of the 1999 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. WASPAA'99 (Cat. No.99TH8452).

[14]  Marinus M. Boone,et al.  Spatial sound reproduction with wave field synthesis , 1999 .

[15]  Michael Williams,et al.  The Quick Reference Guide to Multichannel Microphone Arrays - Part I : using Cardioid Microphones , 2001 .

[16]  Hiroshi Mizoguchi,et al.  Visually steerable sound beam forming method possible to track target person by real-time visual face tracking and speaker array , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[17]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[18]  Lucas C. Parra,et al.  Steerable frequency-invariant beamforming for arbitrary arrays , 2006 .

[19]  Soledad Torres-Guijarro,et al.  On Building Immersive Audio Applications Using Robust Adaptive Beamforming and Joint Audio-Video Source Localization , 2006, EURASIP J. Adv. Signal Process..

[20]  Kiran Jude Fernandes,et al.  Cybersphere: the fully immersive spherical projection system , 2003, CACM.

[21]  Mark A. Poletti,et al.  A Unified Theory of Horizontal Holographic Sound Systems , 2000 .

[22]  Jörg Wuttke.  Surround Recording of Music: Problems and Solutions , 2005 .

[23]  Jerome Daniel,et al.  Further Investigations of High-Order Ambisonics and Wavefield Synthesis for Holophonic Sound Imaging , 2003 .

[24]  Sascha Spors,et al.  Eigenspace adaptive filtering for efficient pre-equalization of acoustic MIMO systems , 2006, 2006 14th European Signal Processing Conference.