3D sound and 3D image interactions: a review of audio-visual depth perception

There has been much research concerning visual depth perception in 3D stereoscopic displays and, to a lesser extent, auditory depth perception in 3D spatial sound systems. With 3D sound systems now available in a number of different forms, there is increasing interest in the integration of 3D sound systems with 3D displays. It therefore seems timely to review key concepts and results concerning depth perception in such display systems. We first present overviews of both visual and auditory depth perception, before focussing on cross-modal effects in audio-visual depth perception, which may be of direct interest to display and content designers.

[1]  Robert S. Allison,et al.  The effect of crosstalk on depth magnitude in thin structures , 2011, Electronic Imaging.

[2]  James J. Clark,et al.  Data Fusion for Sensory Information Processing Systems , 1990 .

[3]  David R Moore,et al.  Auditory perception: The near and far of sound localization , 1999, Current Biology.

[4]  S. Shimojo,et al.  Visual illusion induced by sound. , 2002, Brain research. Cognitive brain research.

[5]  Graham R Jones,et al.  Controlling perceived depth in stereoscopic images , 2001, IS&T/SPIE Electronic Imaging.

[6]  Frank Boland,et al.  Depth perception of audio sources in stereo 3D environments , 2013, Electronic Imaging.

[7]  Tammo Houtgast,et al.  Auditory distance perception in rooms , 1999, Nature.

[8]  R. V. L. Hartley,et al.  The Binaural Location of Pure Tones. , 1921 .

[9]  Glenn S. Smith.  Human color vision and the unsaturated blue color of the daytime sky , 2005 .

[10]  Qp. Bronkhorst,et al.  MODELING AUDITORY DISTANCE PERCEPTION IN ROOMS , 2002 .

[11]  P Zahorik,et al.  Estimating Sound Source Distance with and without Vision , 2001, Optometry and vision science : official publication of the American Academy of Optometry.

[12]  A. Berkhout,et al.  Acoustic control by wave field synthesis , 1993 .

[13]  Michael R. M. Jenkin,et al.  Virtual Audio Systems , 2008, PRESENCE: Teleoperators and Virtual Environments.

[14]  R. Blake,et al.  How Contrast Affects Stereoacuity , 1988, Perception.

[15]  R. Patterson,et al.  Human Stereopsis , 1992, Human factors.

[16]  Søren H. Nielsen.  Depth Perception-Finding a Design Goal for Sound Reproduction Systems , 1991 .

[17]  T Z Strybel,et al.  Discrimination of relative distance in the auditory modality: the success and failure of the loudness discrimination hypothesis. , 1984, The Journal of the Acoustical Society of America.

[18]  G Stigmar,et al.  BLURRED VISUAL STIMULI , 1971 .

[19]  Wa Wijnand IJsselsteijn,et al.  Visual discomfort in stereoscopic displays: a review , 2007, Electronic Imaging.

[20]  N. Langlands,et al.  Experiments on binocular vision. , 1926 .

[21]  Daniel H. Ashmead,et al.  Perception of the relative distances of nearby sound sources , 1990, Perception & psychophysics.

[22]  A. Bronkhorst,et al.  Auditory distance perception in humans : A summary of past and present research , 2005 .

[23]  S. McKee,et al.  Stereoscopic acuity with defocused and spatially filtered retinal images , 1980 .

[24]  H. Levitt.  Transformed up-down methods in psychoacoustics. , 1971, The Journal of the Acoustical Society of America.

[25]  Bliss M. Altenhoff,et al.  Near-field distance perception in real and virtual environments using both verbal and action responses , 2011, TAP.

[26]  Robert A Jacobs,et al.  Bayesian integration of visual and auditory signals for spatial localization. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[27]  P. Coleman.  An analysis of cues to auditory depth perception in free space. , 1963, Psychological bulletin.

[28]  R. Hamer,et al.  Effects of age and optical blur on real depth stereoacuity , 2010, Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians.

[29]  Davide Rocchesso,et al.  Auditory distance perception in an acoustic pipe , 2008, TAP.

[30]  P D Coleman,et al.  Dual role of frequency spectrum in determination of auditory distance. , 1968, The Journal of the Acoustical Society of America.

[31]  A Väljamäe,et al.  Filling-in visual motion with sounds. , 2008, Acta psychologica.

[32]  A. M. Rohaly,et al.  The effects of contrast on perceived depth and depth discrimination , 1999, Vision Research.

[33]  W E Simpson,et al.  Head movement does not facilitate perception of the distance of a source of sound. , 1973, The American journal of psychology.

[34]  G. A. Miller,et al.  Sensitivity to Changes in the Intensity of White Noise and Its Relation to Masking and Loudness , 1947 .

[35]  Neil A. Dodgson,et al.  Three-Dimensional Displays: A Review and Applications Analysis , 2011, IEEE Transactions on Broadcasting.

[36]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[37]  Takashi Okamoto,et al.  Cosmic cookery: making a stereoscopic 3D animated movie , 2006, Electronic Imaging.

[38]  Laurie M Heller,et al.  Auditory — Visual Interactions in the Perception of a Ball's Path , 2005, Perception.

[39]  W. M. Rabinowitz,et al.  Auditory localization of nearby sources. Head-related transfer functions. , 1999, The Journal of the Acoustical Society of America.

[40]  Nick Holliman,et al.  Can the perception of depth in stereoscopic images be influenced by 3D sound? , 2011, Electronic Imaging.

[41]  David M. Hoffman,et al.  The zone of comfort: Predicting visual discomfort with stereo displays. , 2011, Journal of vision.

[42]  Ulrik R Beierholm,et al.  Sound-induced flash illusion as an optimal percept , 2005, Neuroreport.

[43]  Gordon E. Legge,et al.  Stereopsis and contrast , 1989, Vision Research.

[44]  Karen Pfeffer,et al.  Auditory perception of relative distance of traffic sounds , 1998 .

[45]  Adrian David Cheok,et al.  An experimental study on the role of 3D sound in augmented reality environment , 2004, Interact. Comput..

[46]  Y. Yeh,et al.  Limits of Fusion and Depth Judgment in Stereoscopic Color Displays , 1990, Human factors.

[47]  J. Sjöstrand,et al.  Screening merits of the Lang II, Frisby, Randot, Titmus, and TNO stereo tests. , 2001, Journal of AAPOS : the official publication of the American Association for Pediatric Ophthalmology and Strabismus.

[48]  J. Moran,et al.  Sensation and perception , 1980 .

[49]  H J Howard,et al.  A Test for the Judgment of Distance. , 1919, Transactions of the American Ophthalmological Society.

[50]  Andrew Hogue,et al.  The effects of audio on depth perception in S3D games , 2012, AM '12.

[51]  G. Recanzone.  Interactions of auditory and visual stimuli in space and time , 2009, Hearing Research.

[52]  G C DeAngelis,et al.  The physiology of stereopsis. , 2001, Annual review of neuroscience.

[53]  Hugo Fastl,et al.  Minimum Audible Distance (MAD) by the Example of Wave Field Synthesis , 2012 .

[54]  J. Rice.  Mathematical Statistics and Data Analysis , 1988 .

[55]  Pavel Zahorik,et al.  Assessing auditory distance perception using virtual acoustics. , 2002, The Journal of the Acoustical Society of America.

[56]  Jack M. Loomis,et al.  Auditory distance perception by translating observers , 1993, Proceedings of 1993 IEEE Research Properties in Virtual Reality Symposium.

[57]  R. Duda,et al.  Range dependence of the response of a spherical head model , 1998 .

[58]  M. Wallace,et al.  Visual Localization Ability Influences Cross-Modal Bias , 2003, Journal of Cognitive Neuroscience.

[59]  L. Shams,et al.  Crossmodal influences on visual perception. , 2010, Physics of life reviews.

[60]  M. Gardner.  Proximity image effect in sound localization. , 1968, The Journal of the Acoustical Society of America.

[61]  P. Schmidt,et al.  Sensitivity of Random Dot Stereoacuity and Snellen Acuity to Optical Blur , 1994, Optometry and vision science : official publication of the American Academy of Optometry.

[62]  Andrew Hogue,et al.  The effects of 5.1 sound presentations on the perception of stereoscopic imagery in video games , 2013, Electronic Imaging.

[63]  B. Julesz.  Foundations of Cyclopean Perception , 1971 .

[64]  Kirsten Cater,et al.  An investigation into the use of spatialised sound in locative games , 2007, CHI Extended Abstracts.

[65]  P. Coleman.  Failure to Localize the Source Distance of an Unfamiliar Sound , 1962 .

[66]  A. S. Edwards,et al.  Accuracy of Auditory Depth Perception , 1955 .