Acoustic Rendering and Auditory–Visual Cross‐Modal Perception and Interaction

In recent years research in the 3-Dimensional sound generation field has been primarily focussed upon new applications of spatialised sound. In the computer graphics community the use of such techniques is most commonly found being applied to virtual, immersive environments. However, the field is more varied and diverse than this and other research tackles the problem in a more complete, and computationally expensive manner. However, simulation of light and sound wave propagation is still unachievable at a physically accurate spatio-temporal quality in real-time. Although the Human Visual System (HVS) and the Human Auditory System (HAS) are exceptionally sophisticated, they also contain certain perceptional and attentional limitations. Researchers, in fields such as psychology, have been investigating these limitations for several years and have come up with some findings which may be exploited in other fields. This STAR provides a comprehensive overview of the major techniques for generating spatialised sound and, in addition, discusses perceptual and cross-modal influences to consider. We also describe current limitations and provide an in-depth look at the emerging topics in the field.

[1]  J Driver,et al.  Cross-modal selective attention: On the difficulty of ignoring sounds at the locus of visual attention , 2000, Perception & psychophysics.

[2]  Kurt Debattista,et al.  Exploiting Audio-Visual Cross-Modal Interaction to Reduce Computational Requirements in Interactive Environments , 2010, 2010 Second International Conference on Games and Virtual Worlds for Serious Applications.

[3]  Chiew Tong Lau,et al.  The Significance of Tonality Index and Non-linear Psychoacoustics Models for Masking Threshold Estimation , 2002 .

[4]  D R Perrott,et al.  Minimum audible angle thresholds for sources varying in both elevation and azimuth. , 1990, The Journal of the Acoustical Society of America.

[5]  Axel Pinz,et al.  Human Visual System , 2002 .

[6]  Dani Lischinski,et al.  Synthesis of Sound Textures by Learning and Resampling of Wavelet Trees , .

[7]  Remy Bruno,et al.  High Spatial Resolution Multichannel Recording , 2004 .

[8]  Davide Rocchesso,et al.  An introductory catalog of computer-synthesized contact sounds , 2003 .

[9]  W. Hartmann Localization of sound in rooms. , 1983, The Journal of the Acoustical Society of America.

[10]  Stephen Handel Thinking in Sound: The Cognitive Psychology of Human Audition Stephen McAdams Emmanuel Bigand , 1995 .

[11]  Luís Paulo Santos,et al.  Selective rendering: computing only what you see , 2006, GRAPHITE '06.

[12]  George Drettakis,et al.  Fast modal sounds with scalable frequency-domain synthesis , 2008, ACM Trans. Graph..

[13]  A. Berkhout,et al.  Acoustic control by wave field synthesis , 1993 .

[14]  Elizabeth M. Wenzel,et al.  LATENCY MEASUREMENT OF A REAL-TIME VIRTUAL ACOUSTIC ENVIRONMENT RENDERING SYSTEM , 2003 .

[15]  Lester C. Loschky,et al.  User performance with gaze contingent multiresolutional displays , 2000, ETRA.

[16]  Karlheinz Blankenbach Spatial Effects , 2012, Handbook of Visual Display Technology.

[17]  Hans Hagen,et al.  Phonon tracing for auralization and visualization of sound , 2005, VIS 05. IEEE Visualization, 2005..

[18]  Béatrice de Gelder,et al.  A Visual Influence in the Discrimination of Auditory Location , 1998, AVSP.

[19]  David Alais,et al.  Separate attentional resources for vision and audition , 2006, Proceedings of the Royal Society B: Biological Sciences.

[20]  F. Ihlenburg Finite Element Analysis of Acoustic Scattering , 1998 .

[21]  Jyri Huopaniemi,et al.  Real-Time Virtual Audio Reality , 1996, ICMC.

[22]  Jean-Marc Jot,et al.  Digital Signal Processing Issues in the Context of Binaural and Transaural Stereophony , 1995 .

[23]  David G. Malham,et al.  3-D Sound Spatialization using Ambisonic Techniques , 1995 .

[24]  H. Lehnert Systematic errors of the ray-tracing algorithm , 1993 .

[25]  A. Rajkumar,et al.  Predicting RF coverage in large environments using ray-beam tracing and partitioning tree represented geometry , 1996, Wirel. Networks.

[26]  Yoshinori Dobashi,et al.  Synthesizing Sound from Turbulent Field using Sound Textures for Interactive Fluid Simulation , 2004, Comput. Graph. Forum.

[27]  Juha Merimaa,et al.  Applications of a 3-D Microphone Array , 2002 .

[28]  D. Meyer,et al.  Attention and Performance XIV , 1973 .

[29]  Daniel Västfjäll,et al.  Better Presence and Performance in Virtual Environments by Improved Binaural Sound Rendering , 2002 .

[30]  T SHIPLEY,et al.  Auditory Flutter-Driving of Visual Flicker , 1964, Science.

[31]  Özgür Yilmaz,et al.  Blind separation of disjoint orthogonal signals: demixing N sources from 2 mixtures , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[32]  Dinesh Manocha,et al.  Interactive sound rendering , 2009, CAD/Graphics.

[33]  Carlo Harvey,et al.  The Effect of Discretised and Fully Converged Spatialised Sound on Directional Attention and Distraction , 2010, TPCG.

[34]  James F. O'Brien,et al.  Synthesizing Sounds from Physically Based Motion , 2001, SIGGRAPH Video Review on Animation Theater Program.

[35]  Kurt Debattista,et al.  Maintaining frame rate perception in interactive environments by exploiting audio-visual cross-modal interaction , 2010, The Visual Computer.

[36]  Jan Kautz,et al.  Is accurate occlusion of glossy reflections necessary? , 2007, APGV.

[37]  D. Allport,et al.  On the Division of Attention: A Disproof of the Single Channel Hypothesis , 1972, The Quarterly journal of experimental psychology.

[38]  Douglas S. Brungart,et al.  Localization in the presence of multiple simultaneous sounds , 2005 .

[39]  A. Farina RAMSETE-A NEW PYRAMID TRACER FOR MEDIUM AND LARGE SCALE ACOUSTIC PROBLEMS , 2000 .

[40]  V. Bruce,et al.  Visual Cognition: Computational, Experimental, and Neuropsychological Perspectives , 1989 .

[41]  Dinesh K. Pai,et al.  FoleyAutomatic: physically-based sound effects for interactive simulation and animation , 2001, SIGGRAPH.

[42]  C. Spence,et al.  Crossmodal Space and Crossmodal Attention , 2004 .

[43]  Juha Merimaa,et al.  Spatial Impulse Response Rendering , 2004 .

[44]  Sonny Chan,et al.  Sound synthesis for the Web, games, and virtual reality , 2003, SIGGRAPH '03.

[45]  David L. Strayer,et al.  Driven to Distraction: Dual-Task Studies of Simulated Driving and Conversing on a Cellular Telephone , 2001, Psychological science.

[46]  Sophia Antipolis,et al.  SCALABLE PERCEPTUAL MIXING AND FILTERING OF AUDIO SIGNALS USING AN AUGMENTED SPECTRAL REPRESENTATION , 2005 .

[47]  Scott Rickard,et al.  Blind separation of speech mixtures via time-frequency masking , 2004, IEEE Transactions on Signal Processing.

[48]  Emmanuel Gallo,et al.  Extracting and Re-Rendering Structured Auditory Scenes from Field Recordings , 2007 .

[49]  Steven K. Feiner,et al.  Computer graphics: principles and practice (2nd ed.) , 1990 .

[50]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[51]  Samuli Laine,et al.  Accelerated beam tracing algorithm , 2009 .

[52]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[53]  Kurt Debattista,et al.  Level of Realism for Serious Games , 2009, 2009 Conference in Games and Virtual Worlds for Serious Applications.

[54]  Chen Shen,et al.  Synthesizing sounds from rigid-body simulations , 2002, SCA '02.

[55]  Manfred R. Schroeder,et al.  Natural Sounding Artificial Reverberation , 1962 .

[56]  Bernard D. Adelstein,et al.  Head Tracking Latency in Virtual Environments: Psychophysics and a Model , 2003 .

[57]  Michael Wimmer,et al.  Efficient and practical audio-visual rendering for games using crossmodal perception , 2009, I3D '09.

[58]  Durand R. Begault,et al.  Sensitivity to haptic-audio asynchrony , 2003, ICMI '03.

[59]  Gregory B. Newby,et al.  Virtual reality: Scientific and technological challenges , 1996 .

[60]  A. Bonnel,et al.  Divided attention between simultaneous auditory and visual signals , 1998, Perception & psychophysics.

[61]  S. Shimojo,et al.  Illusions: What you see is what you hear , 2000, Nature.

[62]  Dominic W. Massaro,et al.  Dividing attention between auditory and visual perception , 1977 .

[63]  G H MOWBRAY,et al.  On discriminating the rate of visual flicker and auditory flutter. , 1959, The American journal of psychology.

[64]  Veronica Sundstedt,et al.  Visual attention for efficient high-fidelity graphics , 2005, SCCG '05.

[65]  Michael Zyda,et al.  Exploiting reality with multicast groups: a network architecture for large-scale virtual environments , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[66]  J. Borish Extension of the image model to arbitrary polyhedra , 1984 .

[67]  Dinesh Manocha,et al.  Sounding liquids: Automatic sound synthesis from fluid simulation , 2010, TOGS.

[68]  Kavita Bala,et al.  Perception of complex aggregates , 2008, ACM Trans. Graph..

[69]  R. Steinman,et al.  Phi is not beta, and why Wertheimer’s discovery launched the Gestalt revolution , 2000, Vision Research.

[70]  Søren H. Nielsen,et al.  Auditory Distance Perception in Different Rooms , 1993 .

[71]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[72]  Eric Horvitz,et al.  Perception, Attention, and Resources: A Decision-Theoretic Approach to Graphics Rendering , 1997, UAI.

[73]  Mathieu Lagrange,et al.  Real-Time Additive Synthesis of Sound by Taking Advantage of Psychoacoustics , 2001 .

[74]  David G. Malham Spherical Harmonic Coding of Sound Objects - the Ambisonic 'O' Format , 2001 .

[75]  Michael T. Lippert,et al.  Mechanisms for Allocating Auditory Attention: An Auditory Saliency Map , 2005, Current Biology.

[76]  Kurt Debattista,et al.  A GPU based saliency map for high-fidelity selective rendering , 2006, AFRIGRAPH '06.

[77]  RICHARD RADKE,et al.  Radke and Rickard Audio Interpolation AUDIO INTERPOLATION , 2001 .

[78]  David M. Howard,et al.  On the computational efficiency of different waveguide mesh topologies for room acoustic simulation , 2005, IEEE Transactions on Speech and Audio Processing.

[79]  Thomas A. Funkhouser,et al.  Real-time acoustic modeling for distributed virtual environments , 1999, SIGGRAPH.

[80]  Kurt Debattista,et al.  Auditory bias of visual attention for perceptually-guided selective rendering of animations , 2005, GRAPHITE '05.

[81]  Jernej Barbic,et al.  Precomputed acoustic transfer: output-sensitive, accurate sound generation for geometrically complex vibration sources , 2006, ACM Trans. Graph..

[82]  Turner Whitted,et al.  An improved illumination model for shaded display , 1979, CACM.

[83]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[84]  David Lewis Yewdall Practical art of motion picture sound , 2003 .

[85]  Alan Chalmers,et al.  The effect of music on the perception of display rate and duration of animated sequences: an experimental study , 2004, Proceedings Theory and Practice of Computer Graphics, 2004..

[86]  Ming C. Lin,et al.  Precomputed wave simulation for real-time sound propagation of dynamic sources in complex scenes , 2010, ACM Trans. Graph..

[87]  A. L. I︠A︡rbus Eye Movements and Vision , 1967 .

[88]  Daniel P. W. Ellis,et al.  Sound texture modelling with linear prediction in both time and frequency domains , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[89]  Abdellatif Benjelloun Touimi,et al.  Efficient method for multiple compressed audio streams spatialization , 2004, MUM '04.

[90]  André van Schaik,et al.  Auditory spatial perception with sources overlapping in frequency and time , 2005 .

[91]  Nicolas Tsingos,et al.  Perceptually-based auralization , 2007 .

[92]  Ville Pulkki Directional Audio Coding in Spatial Sound Reproduction and Stereo Upmixing , 2006 .

[93]  Qing‐Huo Liu The PSTD algorithm: A time-domain method combining the pseudospectral technique and perfectly matched layers , 1997 .

[94]  J. Theeuwes Exogenous and endogenous control of attention: The effect of visual onsets and offsets , 1991, Perception & psychophysics.

[95]  Ramani Duraiswami,et al.  A broadband fast multipole accelerated boundary element method for the three dimensional Helmholtz equation. , 2009, The Journal of the Acoustical Society of America.

[96]  I. Rock,et al.  Perception without attention: Results of a new method , 1992, Cognitive Psychology.

[97]  Ming C. Lin,et al.  Interactive sound synthesis for large scale environments , 2006, I3D '06.

[98]  Beth Logan,et al.  Mel Frequency Cepstral Coefficients for Music Modeling , 2000, ISMIR.

[99]  Renato S. Pellegrini,et al.  Quality assessment of auditory virtual environments , 2001 .

[100]  Christof Faller,et al.  Binaural cue coding-Part I: psychoacoustic fundamentals and design principles , 2003, IEEE Trans. Speech Audio Process..

[101]  Petros Maragos,et al.  Cross-Modal Integration for Performance Improving in Multimedia: A Review , 2008, Multimodal Processing and Interaction.

[102]  Davide Rocchesso,et al.  Sounding Objects , 2003, IEEE Multim..

[103]  Gerald Westheimer Human spatial orientation: by I. P. Howard and W. B. Templeton. 533 pages, diagrams, illustr., 6×9 in. New York, John Wiley, Inc., 1966. Price, $13.50 , 1967 .

[104]  Shinichi Sakamoto,et al.  Numerical analysis of sound propagation in rooms using the finite difference time domain method , 2006 .

[105]  D. Burr,et al.  Combining visual and auditory information. , 2006, Progress in brain research.

[106]  Remy Bruno,et al.  A New Comprehensive Approach of Surround Sound Recording , 2003 .

[107]  Hiroshi Sawada,et al.  Blind Extraction of Dominant Target Sources Using ICA and Time-Frequency Masking , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[108]  Catharine Abell,et al.  Seeing and Visualizing: It's Not What You Think , 2005 .

[109]  A. Kingstone,et al.  Auditory capture of vision: examining temporal ventriloquism. , 2003, Brain research. Cognitive brain research.

[110]  Bill Kapralos,et al.  GPU-based real-time acoustical occlusion modeling , 2010, Virtual Reality.

[111]  Ag Armin Kohlrausch,et al.  Audio—Visual Interaction in the Context of Multi-Media Applications , 2005 .

[112]  R. Sekuler,et al.  Sound alters visual motion perception , 1997, Nature.

[113]  Jens Herder,et al.  Optimization of Sound Spatialization Resource Management through Clustering , 1999 .

[114]  Ming C. Lin,et al.  Accelerated wave-based acoustics simulation , 2008, SPM '08.

[115]  Thomas A. Funkhouser,et al.  A beam tracing approach to acoustic modeling for interactive virtual environments , 1998, SIGGRAPH.

[116]  Jens Ahrens,et al.  The Single-layer Potential Approach Applied to Sound Field Synthesis Including Cases of Non-enclosing Distributions of Secondary Sources , 2010 .

[117]  Wm Wil Wagenaars Localization of Sound in a Room with Reflecting Walls , 1989 .

[118]  Ming C. Lin,et al.  Efficient and Accurate Sound Propagation Using Adaptive Rectangular Decomposition , 2009, IEEE Transactions on Visualization and Computer Graphics.

[119]  E. Hobson The Theory of Spherical and Ellipsoidal Harmonics , 1955 .

[120]  Jamie Ward,et al.  Sound-Colour Synaesthesia: to What Extent Does it Use Cross-Modal Mechanisms Common to us All? , 2006, Cortex.

[121]  Wolfgang Straßer,et al.  Multi-resolution sound rendering , 2004, SIGGRAPH '04.

[122]  R. Williams,et al.  Source Decomposition for Vehicle Sound Simulation , 2001 .

[123]  D C Donderi,et al.  The effect of sound on visual apparent movement. , 1983, The American journal of psychology.

[124]  Axel Röbel,et al.  A TENTATIVE TYPOLOGY OF AUDIO SOURCE SEPARATION TASKS , 2003 .

[125]  N. Lalor,et al.  Analysis of interior acoustic fields using the finite element method and the boundary element method , 1995 .

[126]  Krzysztof Marasek,et al.  Computation of Room Acoustics using Programmable Video Hardware , 2004, ICCVG.

[127]  Yoshinori Dobashi,et al.  Real-time rendering of aerodynamic sound using sound textures based on computational fluid dynamics , 2003, ACM Trans. Graph..

[128]  D. C. Higgins Human Spatial Orientation , 1967, The Yale Journal of Biology and Medicine.

[129]  C. Faller,et al.  Source localization in complex listening situations: selection of binaural cues based on interaural coherence. , 2004, The Journal of the Acoustical Society of America.

[130]  D. Botteldooren Finite‐difference time‐domain simulation of low‐frequency room acoustic problems , 1995 .

[131]  Tapio Lokki,et al.  A framework for evaluating virtual acoustics environments , 2001 .

[132]  Lie Lu,et al.  Audio textures: theory and applications , 2004, IEEE Transactions on Speech and Audio Processing.

[133]  J. Blauert Spatial Hearing: The Psychophysics of Human Sound Localization , 1983 .

[134]  Guillaume Lemaitre,et al.  PRIORITIZING SIGNALS FOR SELECTIVE REAL-TIME AUDIO PROCESSING , 2005 .

[135]  Kurt Debattista,et al.  The influence of sound effects on the perceived smoothness of rendered animations , 2005, APGV '05.

[136]  Dinesh Manocha,et al.  An efficient GPU-based time domain solver for the acoustic wave equation , 2012 .

[137]  Claus Bundesen,et al.  Seeing or hearing? Perceptual independence, modality confusions, and crossmodal congruity effects with focused and divided attention , 2003, Perception & psychophysics.

[138]  E C Haas,et al.  Perceived urgency of and response time to multi-tone and frequency-modulated warning signals in broadband noise. , 1995, Ergonomics.

[139]  Doug L. James,et al.  Animating fire with sound , 2011, ACM Trans. Graph..

[140]  Jont B. Allen,et al.  Image method for efficiently simulating small‐room acoustics , 1976 .

[141]  Vesa Välimäki,et al.  Interpolated 3-D digital waveguide mesh with frequency warping , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[142]  Glenn N. Dickins,et al.  Optimal 3D Speaker Panning , 1999 .

[143]  E. Owens,et al.  An Introduction to the Psychology of Hearing , 1997 .

[144]  Alan Chalmers,et al.  Selective quality rendering by exploiting human inattentional blindness: looking but not seeing , 2002, VRST '02.

[145]  Christof Faller,et al.  Binaural cue coding-Part II: Schemes and applications , 2003, IEEE Trans. Speech Audio Process..

[146]  Anthony I. Tew,et al.  The Continuity Illusion in Virtual Auditory Space , 2002 .

[147]  Jürgen Herre,et al.  Perceptual Coding of High-Quality Digital Audio , 2013, Proceedings of the IEEE.

[148]  Michael W. Eysenck Perception and Communication: D.E. Broadbent , 1994 .

[149]  E. H. Linfoot Principles of Optics , 1961 .

[150]  Tapio Lokki,et al.  Creating Interactive Virtual Acoustic Environments , 1999 .

[151]  Doug L. James,et al.  Rigid-body fracture sound with precomputed soundbanks , 2010, ACM Trans. Graph..

[152]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[153]  George Drettakis,et al.  Perceptual audio rendering of complex virtual environments , 2004, ACM Trans. Graph..

[154]  G. Recanzone Auditory influences on visual temporal rate perception. , 2003, Journal of neurophysiology.

[155]  Waka Fujisaki,et al.  Temporal frequency characteristics of synchrony–asynchrony discrimination of audio-visual signals , 2005, Experimental Brain Research.

[156]  S. Shimojo,et al.  Visual illusion induced by sound. , 2002, Brain research. Cognitive brain research.

[157]  Earl Vickers,et al.  Frequency Domain Artificial Reverberation using Spectral Magnitude Decay , 2006 .

[158]  T. Anderson,et al.  Binaural and spatial hearing in real and virtual environments , 1997 .

[159]  E. Milios,et al.  Sonel mapping: acoustic modeling utilizing an acoustic version of photon mapping , 2004, Proceedings. Second International Conference on Creating, Connecting and Collaborating through Computing.

[160]  Dirk Bartz,et al.  The Role of Perception for Computer Graphics , 2008, Eurographics.

[161]  Arthur Appel,et al.  Some techniques for shading machine renderings of solids , 1968, AFIPS Spring Joint Computing Conference.

[162]  Alan Chalmers,et al.  Detail to Attention: Exploiting Visual Tasks for Selective Rendering , 2003, Rendering Techniques.

[163]  Randolph Blake,et al.  Hearing What the Eyes See , 2005, Psychological science.

[164]  Jan Theeuwes,et al.  Pip and pop: nonspatial auditory signals improve spatial visual search. , 2008, Journal of experimental psychology. Human perception and performance.

[165]  A. Krokstad,et al.  Calculating the acoustical room response by the use of a ray tracing technique , 1968 .

[166]  André Dufour,et al.  Importance of attentional mechanisms in audiovisual links , 1999, Experimental Brain Research.

[167]  Durand R. Begault,et al.  3-D Sound for Virtual Reality and Multimedia Cambridge , 1994 .

[168]  Graham Naylor,et al.  ODEON—Another hybrid room acoustical model , 1993 .

[169]  Robert B. Welch,et al.  The “ventriloquist effect”: Visual dominance or response bias? , 1975 .

[170]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[171]  Gary W. Elko,et al.  Spherical Microphone Arrays for 3D Sound Recording , 2004 .

[172]  Jon Driver,et al.  Crossmodal attention , 1998, Current Opinion in Neurobiology.

[173]  Georgia Mastoropoulou The effect of audio on the visual perception of high-fidelity animated 3D computer graphics , 2007 .

[174]  D. Spalding The Principles of Psychology , 1873, Nature.

[175]  Juha Merimaa,et al.  Spatial Impulse Response Rendering I: Analysis and Synthesis , 2005 .

[176]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[177]  Daniel J. Levitino,et al.  The Perception of Cross-Modal Simultaneity , 2001 .

[178]  Kees van den Doel,et al.  Physically based models for liquid sounds , 2005, TAP.

[179]  Russell L. Storms Auditory-visual cross-modal perception phenomena , 1998 .

[180]  D. Burr,et al.  Auditory dominance over vision in the perception of interval duration , 2009, Experimental Brain Research.

[181]  E. Bruce Goldstein,et al.  Encyclopedia of perception , 2010 .

[182]  Peter Mark Roget V. Explanation of an optical deception in the appearance of the spokes of a wheel seen through vertical apertures , 1825, Philosophical Transactions of the Royal Society of London.

[183]  France Télécom A GENERIC FRAMEWORK FOR FILTERING IN SUBBAND-DOMAIN Abdellatif Benjelloun Touimi , 2000 .

[184]  David M. Howard,et al.  Waveguide physical modeling of vocal tract acoustics: flexible formant bandwidth control from increased model dimensionality , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[185]  Soto-Faraco Salvador,et al.  AUDIO-VISUAL INTERACTIONS IN DYNAMIC SCENES: IMPLICATIONS FOR MULTISENSORY COMPRESSION , 2007 .

[186]  Doug L. James,et al.  Toward high-quality modal contact sound , 2011, ACM Trans. Graph..

[187]  Agostino Di Scipio,et al.  SYNTHESIS OF ENVIRONMENTAL SOUND TEXTURES BY ITERATED NONLINEAR FUNCTIONS , 1999 .

[188]  Donald P. Greenberg,et al.  Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments , 2005, TOGS.

[189]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[190]  SCRIME-LaBRI STATISTICAL APPROACH FOR SOUND MODELING , 2000 .

[191]  S. Shimojo,et al.  Sensory modalities are not separate modalities: plasticity and interactions , 2001, Current Opinion in Neurobiology.

[192]  Woodrow Barfield,et al.  Presence within Virtual Environments as a Function of Visual Display Parameters , 1996, Presence: Teleoperators & Virtual Environments.

[193]  Pat Hanrahan,et al.  Beam tracing polygonal objects , 1984, SIGGRAPH.

[194]  Guillaume Lemaitre,et al.  3D-Audio Matting, Postediting, and Rerendering from Field Recordings , 2007, EURASIP J. Adv. Signal Process..

[195]  T. Ajdler,et al.  The Plenacoustic Function and Its Sampling , 2006, IEEE Transactions on Signal Processing.

[196]  George Drettakis,et al.  Bimodal perception of audio-visual material properties for virtual environments , 2010, TAP.

[197]  B. Moore An introduction to the psychology of hearing, 3rd ed. , 1989 .

[198]  Mark Sandler,et al.  DIGITAL AUDIO EFFECTS IN THE WAVELET DOMAIN , 2002 .

[199]  C. Avendano,et al.  Frequency-domain source identification and manipulation in stereo mixes for enhancement, suppression and re-panning applications , 2003, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684).

[200]  Michael S. Lewicki,et al.  Efficient coding of natural sounds , 2002, Nature Neuroscience.

[201]  Ian Burnett,et al.  DECORRELATION TECHNIQUES FOR THE RENDERING OF APPARENT SOUND SOURCE WIDTH IN 3D AUDIO DISPLAYS , 2004 .

[202]  Dick Botteldooren,et al.  ACOUSTICAL FINITE-DIFFERENCE TIME-DOMAIN SIMULATION IN A QUASI-CARTESIAN GRID , 1994 .

[203]  Hideki Tachibana,et al.  Calculation of impulse responses and acoustic parameters in a hall by the finite-difference time-domain method , 2008 .

[204]  Ana Tajadura-Jiménez,et al.  Perceptual Optimization of Audio-visual Media: Moved by sound. , 2007 .

[205]  Alan Chalmers,et al.  The influence of cross-modal interaction on perceived rendering quality thresholds , 2008 .

[206]  Dani Lischinski,et al.  Granular Synthesis of Sound Textures Using Statistical Learning , 1999, ICMC.

[207]  D. Murphy,et al.  Acoustic Modeling Using the Digital Waveguide Mesh , 2007, IEEE Signal Processing Magazine.

[208]  Albert S. Bregman,et al.  The Auditory Scene. (Book Reviews: Auditory Scene Analysis. The Perceptual Organization of Sound.) , 1990 .

[209]  Xavier Serra,et al.  Audio Descriptors and Descriptor Schemes in the Context of MPEG-7 , 1999, ICMC.

[210]  C. Chabris,et al.  Gorillas in Our Midst: Sustained Inattentional Blindness for Dynamic Events , 1999, Perception.

[211]  James M. Calvin,et al.  The SIMNET virtual world architecture , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[212]  G. Aschersleben,et al.  Temporal ventriloquism: crossmodal interaction on the time dimension. 1. Evidence from auditory-visual temporal order judgment. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[213]  Rachel McDonnell,et al.  Perceptually Adaptive Graphics , 2004, Eurographics.

[214]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[215]  Dinesh Manocha,et al.  Use of GPUs in room acoustic modeling and auralization , 2010 .

[216]  Dinesh K. Pai,et al.  Interactive Simulation of Complex Audiovisual Scenes , 2004, Presence: Teleoperators & Virtual Environments.

[217]  George Drettakis,et al.  Progressive perceptual audio rendering of complex scenes , 2007, SI3D.

[218]  Jean-Marc Jot,et al.  Real-time spatial processing of sounds for music, multimedia and interactive human-computer interfaces , 1999, Multimedia Systems.

[219]  Michael A. Gerzon,et al.  Ambisonics in Multichannel Broadcasting and Video , 1985 .

[220]  B. Scholl Objects and attention: the state of the art , 2001, Cognition.

[221]  Dinesh Manocha,et al.  Real-time sound synthesis and propagation for games , 2007, CACM.

[222]  Wolfgang Ahnert,et al.  EARS auralization software , 1993 .

[223]  Maic Masuch,et al.  RAY ACOUSTICS USING COMPUTER GRAPHICS TECHNOLOGY , 2007 .

[224]  Thushara D. Abhayapala,et al.  Theory and design of high order sound field microphones using spherical microphone array , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[225]  Dinesh K. Pai,et al.  Precomputed acoustic transfer: output-sensitive, accurate sound generation for geometrically complex vibration sources , 2006, SIGGRAPH 2006.

[226]  Sylvain Lefebvre,et al.  Instant Sound Scattering , 2007, Rendering Techniques.

[227]  Dylan Menzies W-Panning and O-Format, Tools for Object Spatialization , 2002 .

[228]  J Edworthy,et al.  Improving Auditory Warning Design: Relationship between Warning Sound Parameters and Perceived Urgency , 1991, Human factors.

[229]  Ville Pulkki,et al.  Virtual Sound Source Positioning Using Vector Base Amplitude Panning , 1997 .

[230]  Bruce Walter,et al.  Visual equivalence: towards a new standard for image fidelity , 2007, SIGGRAPH 2007.

[231]  Jerome Daniel,et al.  Ambisonics Encoding of Other Audio Formats for Multiple Listening Conditions , 1998 .

[232]  Joseph D. Anderson,et al.  The myth of persistence of vision revisited , 1993 .

[233]  Henrik Wann Jensen,et al.  Global Illumination using Photon Maps , 1996, Rendering Techniques.

[234]  Barak A. Pearlmutter,et al.  Survey of sparse and non‐sparse methods in source separation , 2005, Int. J. Imaging Syst. Technol..

[235]  Minh N. Do Toward sound-based synthesis: the far-field case , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[236]  Jacob Benesty,et al.  Audio Signal Processing for Next-Generation Multimedia Communication Systems , 2004 .

[237]  Henrik Møller Reproduction of artificial head recordings through loudspeakers , 1989 .

[238]  J. Weisenberger Fundamentals of Hearing: An Introduction (3rd ed.) , 1994 .

[239]  Henrik Møller Fundamentals of binaural technology , 1991 .

[240]  Charles Q. Robinson,et al.  Surround Sound with Height in Games Using Dolby Pro Logic IIz , 2010 .

[241]  Doug L. James,et al.  Harmonic fluids , 2009, ACM Trans. Graph..

[242]  Stephan Getzmann,et al.  The Effect of Brief Auditory Stimuli on Visual Apparent Motion , 2007, Perception.

[243]  James T. Kajiya,et al.  The rendering equation , 1986, SIGGRAPH.

[244]  B. Stein,et al.  Enhancement of Perceived Visual Intensity by Auditory Stimuli: A Psychophysical Analysis , 1996, Journal of Cognitive Neuroscience.

[245]  Thomas A. Funkhouser,et al.  Modeling acoustics in virtual environments using the uniform theory of diffraction , 2001, SIGGRAPH.

[246]  C. Spence,et al.  Perceptual effects of cross-modal stimulation , 2022 .

[247]  Dinesh K. Pai,et al.  MEASUREMENTS OF PERCEPTUAL QUALITY OF CONTACT SOUND MODELS , 2002 .

[248]  Tapio Lokki,et al.  The room acoustic rendering equation. , 2007, The Journal of the Acoustical Society of America.

[249]  Wenyu Jiang,et al.  Using Programmable Graphics Hardware for Acoustics and Audio Rendering , 2009 .

[250]  Scott T. Rickard,et al.  Sparse sources are separated sources , 2006, 2006 14th European Signal Processing Conference.

[251]  Véronique Larcher,et al.  Techniques de spatialisation des sons pour la réalité virtuelle , 2001 .

[252]  Norimichi Kitagawa,et al.  Audio-visual integration in temporal perception. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[253]  Jean-Marc Jot,et al.  A Comparative Study of 3-D Audio Encoding and Rendering Techniques , 1999 .

[254]  Kurt Debattista,et al.  Investigation of the beat rate effect on frame rate for animated content , 2009, SCCG.

[255]  Paul Bertelson,et al.  Temporal ventriloquism: crossmodal interaction on the time dimension. 2. Evidence from sensorimotor synchronization. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[256]  H. Pashler The Psychology of Attention , 1997 .

[257]  Ken-ichi Anjyo,et al.  Tour into the picture: using a spidery mesh interface to make animation from a single image , 1997, SIGGRAPH.

[258]  R M Warren,et al.  Illusory continuity of tonal and infratonal periodic sounds. , 1988, The Journal of the Acoustical Society of America.