Realtime audiovisual rendering and contemporary audiovisual art

Visual rendering is the process of creating synthetic images of digital models. The modelling of sound synthesis and propagation in a virtual space is called sound rendering. In this article we review different audiovisual rendering techniques suitable for realtime rendering of three-dimensional virtual worlds. Virtual environments are useful in various application areas, for example in architectural visualisation. With audiovisual rendering, lighting and acoustics of a modelled concert hall can be experienced early in the design stage of the building. In this article we demonstrate an interactive audiovisual rendering system where an animated virtual orchestra plays in a modelled concert hall. Virtual musicians are conducted by a real conductor who wears a wired data dress suit and a baton. The conductor and the audience hear the music rendered according to the acoustics of the virtual concert hall, creating a lifelike experience.

[1]  Julius O. Smith,et al.  Efficient Simulation of the Reed-Bore and Bow-String Mechanisms , 1986, ICMC.

[2]  Bernard D. Adelstein,et al.  Factors influencing operator interaction with virtual objects viewed via head-mounted see-through displays: viewing conditions and rendering latency , 1997, Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality.

[3]  Julius O. Smith,et al.  Efficient Synthesis of Stringed Musical Instruments , 1993, ICMC.

[4]  Matti Karjalainen,et al.  Objective and Subjective Evaluation of Head-Related Transfer Function Filter Design , 1999 .

[5]  Unto K. Laine,et al.  A real-time DSP implementation of a flute model , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[6]  Durand R. Begault,et al.  3-D Sound for Virtual Reality and Multimedia Cambridge , 1994 .

[7]  Raúl Rojas,et al.  Neural Networks - A Systematic Introduction , 1996 .

[8]  Julius O. Smith,et al.  Music applications of digital waveguides , 1987 .

[9]  Michael Friis Sørensen,et al.  Head-Related Transfer Functions of Human Subjects , 1995 .

[10]  W. G. Gardner,et al.  3-D Audio Using Loudspeakers , 1998 .

[11]  Julius O. Smith.  A new approach to digital reverberation using closed waveguide networks , 1985 .

[12]  A. Krokstad,et al.  Calculating the acoustical room response by the use of a ray tracing technique , 1968 .

[13]  Matti Karjalainen,et al.  Efficient and Parametric Reverberator for Room Acoustics Modeling , 1997, ICMC.

[14]  Tapio Takala.  Virtual orchestra performance , 1997, SIGGRAPH '97.

[15]  Shuji Hashimoto,et al.  A computer music system that follows a human conductor , 1991, Computer.

[16]  Ville Pulkki,et al.  Virtual Sound Source Positioning Using Vector Base Amplitude Panning , 1997 .

[17]  J. Borish.  Extension of the image model to arbitrary polyhedra , 1984 .

[18]  Jont B. Allen,et al.  Image method for efficiently simulating small‐room acoustics , 1976 .

[19]  Mendel Kleiner,et al.  Auralization-An Overview , 1993 .

[20]  Vesa Välimäki,et al.  Virtual musical instruments — natural sound using physical models , 1996, Organised Sound.

[21]  Mendel Kleiner,et al.  Experimental auralization of car audio installations , 1996 .

[22]  Unto K. Laine,et al.  Transmission-Line Modeling and Real-Time Synthesis of String and Wind Instruments , 1991, ICMC.

[23]  Perry R. Cook,et al.  TBone: An Interactive WaveGuide Brass Instrument Synthesis Workbench for the NeXT Machine , 1991, ICMC.

[24]  Tapio Takala,et al.  Sound rendering , 1992, SIGGRAPH.

[25]  Gary P. Scavone.  Modeling and Control of Performance Expressin in Digital Waveguide Models of Woodwind Instruments , 1996, ICMC.

[26]  Julius O. Smith,et al.  Performance Expression in Commuted Waveguide Synthesis of Bowed Strings , 1995, ICMC.

[27]  Matti Karjalainen,et al.  Modeling of Woodwind Bores with Finger Holes , 1993, ICMC.

[28]  Andrew S. Glassner,et al.  Principles of Digital Image Synthesis , 1995 .

[29]  Matti Karjalainen,et al.  Modeling of reflections and air absorption in acoustical spaces a digital filter design approach , 1997, Proceedings of 1997 Workshop on Applications of Signal Processing to Audio and Acoustics.

[30]  Matti Karjalainen,et al.  Plucked-string models: From the Karplus-Strong algorithm to digital waveguides and beyond , 1998 .

[31]  Tapio Takala,et al.  Simulation of Room Acoustics with a 3-D Finite Difference Mesh , 1994, ICMC.

[32]  Peter Astheimer.  Acoustic Simulation for Visualization and Virtual Reality , 1995 .

[33]  Ken Perlin,et al.  Improv: a system for scripting interactive actors in virtual worlds , 1996, SIGGRAPH.

[34]  Vesa Välimäki,et al.  Development and Calibration of a Guitar Synthesizer , 1997 .

[35]  Kevin Karplus,et al.  Digital Synthesis of Plucked-String and Drum Timbers , 1983 .

[36]  Vesa Välimäki,et al.  Physical Modeling of Plucked String Instruments with Application to Real-Time Sound Synthesis , 1996 .

[37]  Henrik Møller.  Fundamentals of binaural technology , 1991 .

[38]  Julius O. Smith,et al.  Extensions of the Karplus-Strong Plucked-String Algorithm , 1983 .

[39]  V. Välimäki,et al.  Model-Based Analysis/Synthesis of the Acoustic Guitar , 1994 .

[40]  Tapio Lokki,et al.  Creating Interactive Virtual Acoustic Environments , 1999 .