Towards Realistic Immersive Audiovisual Simulations for Hearing Research: Capture, Virtual Scenes and Reproduction

Most current hearing research laboratories and hearing aid evaluation setups are not sufficient to simulate real-life situations and to evaluate future generations of hearing aids that might include gaze information and brain signals. Thus, new methodologies and technologies might need to be implemented in hearing laboratories and clinics in order to generate audiovisual realistic testing environments. The aim of this work is to provide a comprehensive review of the current available approaches and future directions to create audiovisual realistic immersive simulations for hearing research. Additionally, we present the technologies and use cases of our laboratory, as well as the pros and cons of such technologies: From creating 3D virtual simulations with computer graphics and virtual acoustic simulations, to 360º videos and Ambisonic recordings.

[1]  Olli Rummukainen.  Reproducing reality: Perception and quality in immersive audiovisual environments , 2016 .

[2]  Gerald Kidd,et al.  Design and preliminary testing of a visually guided hearing aid. , 2013, The Journal of the Acoustical Society of America.

[3]  Anton Nijholt,et al.  Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes , 2001, CHI.

[4]  Michael Vorländer,et al.  Performance Evaluation of a Dynamic Crosstalk-Cancellation System with Compensation of Early Reflections , 2016 .

[5]  John C. Hart,et al.  The CAVE: audio visual experience automatic virtual environment , 1992, CACM.

[6]  Xavier Serra,et al.  Freesound 2: An Improved Platform for Sharing Audio Clips , 2011 .

[7]  Stefan Debener,et al.  Identifying auditory attention with ear-EEG: cEEGrid versus high-density cap-EEG comparison , 2016, Journal of neural engineering.

[8]  Luca Simeone,et al.  3D Tune-In: The Use of 3D Sound and Gamification to Aid Better Adoption of Hearing Aid Technologies , 2016 .

[9]  Peter J. Passmore,et al.  User Experience of Panoramic Video in CAVE-like and Head Mounted Display Viewing Conditions , 2017, TVX.

[10]  Norbert Dillier,et al.  Comparison of Higher-Order Ambisonics, Vector- and Distance-Based Amplitude Panning using a hearing device beamformer , 2017 .

[11]  Ville Pulkki,et al.  Immersive Audiovisual Environment with 3D Audio Playback , 2012 .

[12]  Roel Vertegaal,et al.  The Attentive Hearing Aid: Eye Selection of Auditory Sources for Hearing Impaired Users , 2009, INTERACT.

[13]  Etienne Parizet,et al.  Investigation on localisation accuracy for first and higher order ambisonics reproduced sound sources , 2013 .

[14]  Jerome Daniel,et al.  Further Investigations of High-Order Ambisonics and Wavefield Synthesis for Holophonic Sound Imaging , 2003 .

[15]  Archontis Politis,et al.  Parametric Time-Frequency Domain Spatial Audio , 2017 .

[16]  Giso Grimm,et al.  Spatial Acoustic Scenarios in Multichannel Loudspeaker Systems for Hearing Aid Evaluation. , 2016, Journal of the American Academy of Audiology.

[17]  Archontis Politis,et al.  First‐Order Directional Audio Coding (DirAC) , 2017 .

[18]  Bernd Porr,et al.  Real-time estimation of horizontal gaze angle by saccade integration using in-ear electrooculography , 2018, PloS one.

[19]  Gerard Llorach,et al.  Influence of visual cues on head and eye movements during listening tasks in multi-talker audiovisual environments with animated characters , 2018, Speech Commun..

[20]  Gerard Llorach,et al.  Web-Based Live Speech-Driven Lip-Sync , 2016, 2016 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES).

[21]  Ville Pulkki,et al.  Spatial Sound Reproduction with Directional Audio Coding , 2007 .

[22]  Adrian Hilton,et al.  3D Scene Reconstruction from Multiple Spherical Stereo Pairs , 2013, International Journal of Computer Vision.

[23]  Jont B. Allen,et al.  Image method for efficiently simulating small‐room acoustics , 1976 .

[24]  H. Kayser,et al.  Open signal processing software platform for hearing aid research ( openMHA ) , 2017 .

[25]  Mary T Cord,et al.  Relationship between laboratory measures of directional advantage and everyday success with directional microphone hearing aids. , 2004, Journal of the American Academy of Audiology.

[26]  Jerome Daniel,et al.  Ambisonics Encoding of Other Audio Formats for Multiple Listening Conditions , 1998 .

[27]  G. Grimm,et al.  Virtual acoustic environments for comprehensive evaluation of model-based hearing devices * , 2018, International journal of audiology.

[28]  Tapio Lokki,et al.  Parametric Multidirectional Decomposition of Microphone Recordings for Broadband High-Order Ambisonic Encoding , 2018 .

[29]  Giso Grimm,et al.  Evaluation of spatial audio reproduction schemes for application in hearing aid research , 2015, ArXiv.

[30]  Francis Rumsey,et al.  Whose Head Is it Anyway? Optimizing Binaural Audio , 2011 .

[31]  Sascha Spors,et al.  A Comparison of Wave Field Synthesis and Higher-Order Ambisonics with Respect to Physical Properties and Spatial Sampling , 2008 .

[32]  Bernd Porr,et al.  Real-time estimation of horizontal eye-gaze angle using in-ear electrooculography by saccade integration - Dataset , 2018 .

[33]  Ville Pulkki,et al.  Localization of virtual sources in multichannel audio reproduction , 2005, IEEE Transactions on Speech and Audio Processing.

[34]  Graham Naylor,et al.  ODEON—Another hybrid room acoustical model , 1993 .

[35]  Samuel W. Clapp,et al.  Interactive simulation and free-field auralization of acoustic space with the rtSOFE , 2017 .

[36]  R. Bentler.  Effectiveness of directional microphones and noise reduction schemes in hearing aids: a systematic review of the evidence. , 2005, Journal of the American Academy of Audiology.

[37]  M. Florentine,et al.  Loudness in Daily Environments , 2011 .

[38]  Archontis Politis,et al.  Enhancement of ambisonic binaural reproduction using directional audio coding with optimal adaptive mixing , 2017, 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA).

[39]  A. Berkhout,et al.  Acoustic control by wave field synthesis , 1993 .

[40]  Gerhard Tröster,et al.  Recognition of Hearing Needs from Body and Eye Movements to Improve Hearing Instruments , 2011, Pervasive.

[41]  T. Herzke,et al.  Toolbox for acoustic scene creation and rendering (TASCAR): Render methods and research applications , 2015 .

[42]  Ville Pulkki,et al.  Virtual Sound Source Positioning Using Vector Base Amplitude Panning , 1997 .

[43]  M. Walger,et al.  Validating a Method to Assess Lipreading, Audiovisual Gain, and Integration During Speech Reception With Cochlear-Implanted and Normal-Hearing Subjects Using a Talking Head , 2017, Ear and hearing.