New Aspects of Virtual Sound Source Localization Research–Impact of Visual Angle and 3-D Video Content on Sound Perception

The influence of image on virtual sound source localization, called the “image proximity effect” or the “ventriloquism effect,” is a well-known phenomenon. This paper focuses on other aspects related to this effect, namely the impact of the visual angle of the presented object and 3-D video content on sound perception. The research conducted confirmed that the visual angle of the presented object determines the image proximity effect regardless of the screen size. An interesting observation was made when studying the impact of 3-D video on virtual sound source localization. When two objects are displayed in a 3-D scene, the viewer’s attention is more attracted by the object that is closer to the viewer (negative parallax). Two eyegaze tracking systems were exploited in the presented experiments to objectivize the obtained results.

[1]  Bozena Kostek,et al.  Exploiting Audio-Visual Correlation by Means of Gaze Tracking , 2010, Int. J. Comput. Sci. Appl..

[2]  G. Aschersleben,et al.  Automatic visual bias of perceived auditory location , 1998 .

[3]  Paul Bertelson,et al.  Ventriloquism, sensory interaction, and response bias: Remarks on the paper by Choe, Welch, Gilford, and Juola , 1976 .

[4]  Kaoru Watanabe,et al.  A Method of 3-D Sound Image Localization using Loudspeaker Arrays , 2003 .

[5]  Bozena Kostek,et al.  Objectivization of Audio-Video Correlation Assessment Experiments , 2010 .

[6]  Søren Bech,et al.  Interaction Between Audio-Visual Factors in a Home Theater System: Experimental Results , 1995 .

[7]  M. Ercan Altinsoy,et al.  The Quality of Auditory-Tactile Virtual Environments , 2012 .

[8]  A. Kingstone,et al.  Auditory capture of vision: examining temporal ventriloquism. , 2003, Brain research. Cognitive brain research.

[9]  Francis Rumsey,et al.  Perceptual Enhancement of Wavefield Synthesis by Stereophonic Means , 2007 .

[10]  Sebastian Merchel,et al.  Touch the Sound: Audio-Driven Tactile Feedback for Audio Mixing Applications , 2012 .

[11]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[12]  M. Gardner.  Proximity image effect in sound localization. , 1968, The Journal of the Acoustical Society of America.

[13]  M. F. Fuller,et al.  Practical Nonparametric Statistics; Nonparametric Statistical Inference , 1973 .

[14]  P. Bertelson,et al.  Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations , 1977 .

[15]  Bozena Kostek,et al.  Objectivization of Audio-Visual Correlation analysis , 2012 .

[16]  Francis Rumsey.  Hear, Hear! Psychoacoustics and Subjective Evaluation , 2011 .

[17]  P. Bertelson,et al.  The ventriloquist effect does not depend on the direction of automatic visual attention , 2001, Perception & psychophysics.

[18]  Floyd E. Toole,et al.  Loudspeakers and Rooms for Sound Reproduction-A Scientific Review , 2006 .

[19]  Bernard Mendiburu,et al.  3D Movie Making: Stereoscopic Digital Cinema from Script to Screen , 2009 .

[20]  George Kalliris,et al.  Improved Localization of Sound Sources Using Multi-Band Processing of Ambisonic Components , 2009 .

[21]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[22]  Agnieszka Roginska.  Effect of Spatial Location and Presentation Rate on the Reaction to Auditory Displays , 2012 .

[23]  Francis Rumsey.  Audio in Multimodal Applications , 2010 .

[24]  Kazuho Ono,et al.  Advanced Multichannel Audio Systems with Superior Impression of Presence and Reality , 2004 .

[25]  George Kalliris,et al.  Sound Source Localization and B-Format Enhancement Using Soundfield Microphone Sets , 2007 .

[26]  P. Bertelson,et al.  Cross-modal bias and perceptual fusion with auditory-visual spatial discordance , 1981, Perception & psychophysics.

[27]  C. Spence,et al.  Perceptual effects of cross-modal stimulation , 2022 .

[28]  Francis Rumsey.  Sound Quality Evaluation , 2010 .

[29]  Laurence R. Harris,et al.  Eye position affects the perceived location of touch , 2009, Experimental Brain Research.

[30]  Charalampos Dimoulas,et al.  Live Broadcasting of High Definition Audiovisual Content Using HDTV over Broadband IP Networks , 2008, Int. J. Digit. Multim. Broadcast..

[31]  Masaki Emoto,et al.  Viewing angle effects from wide field video projection images on the human equilibrium , 2005, Displays.

[32]  Larry F. Hodges,et al.  Can Audio Enhance Visual Perception and Performance in a Virtual Environment? , 1999 .

[33]  P. Bertelson,et al.  The ventriloquist effect does not depend on the direction of deliberate visual attention , 2000, Perception & psychophysics.

[34]  Koichiro Hiyama,et al.  Effectiveness of Height Information for Reproducing the Presence and Reality in Multichannel Audio System , 2006 .

[35]  Bozena Kostek.  Perception-Based Data Processing in Acoustics: Applications to Music Information Retrieval and Psychophysiology , 2005, Studies in Computational Intelligence.

[36]  H. A. Witkin,et al.  Sound localization with conflicting visual and auditory cues. , 1952, Journal of experimental psychology.

[37]  Maximo Cobos,et al.  Resynthesis of Sound Scenes on Wave-Field Synthesis from Stereo Mixtures Using Sound Source Separation Algorithms , 2009 .

[38]  Paul Bertelson,et al.  The aftereffects of ventriloquism: generalization across sound-frequencies. , 2005, Acta psychologica.

[39]  Bozena Kostek,et al.  An new method of audio-visual correlation analysis , 2009, 2009 International Multiconference on Computer Science and Information Technology.

[40]  Setsu Komiyama,et al.  Subjective Evaluation of Angular Displacement between Picture and Sound Directions for HDTV Sound Systems , 1989 .

[41]  Michael Peter Hollier,et al.  Multi-modal Perception , 1999 .

[42]  Jürgen Herre,et al.  Interactive Teleconferencing Combining Spatial Audio Object Coding and DirAC Technology , 2010 .

[43]  John G. Beerends,et al.  The Influence of Video Quality on Perceived Audio Quality and Vice Versa , 1999 .

[44]  Jens Blauert,et al.  A Layer Model of Sound Quality , 2012 .