Experiments with Virtual Reality Instruments

In this paper, we introduce and analyze four gesture-controlled musical instruments. We briefly discuss the test platform designed to allow for rapid experimentation of new interfaces and control mappings. We describe our design experiences and discuss the effects of system features such as latency, resolution and lack of tactile feedback. The instruments use virtual reality hardware and computer vision for user input, and three-dimensional stereo vision as well as simple desktop displays for providing visual feedback. The instrument sounds are synthesized in real-time using physical sound modeling.

[1]  Roel Vertegaal,et al.  Comparison of Input Devices in an ISEE Direct Timbre Manipulation Task , 1996, Interact. Comput..

[2]  J. A. Paradiso.  Electronic money: toward a virtual wallet , 1997 .

[3]  Axel G. E. Mulder.  Design of virtual three-dimensional instruments for sound control , 1998 .

[4]  Marcelo M. Wanderley,et al.  Towards a Model for Instrumental Mapping in Expert Musical Interaction , 2000, ICMC.

[5]  Insook Choi,et al.  A Manifold Interface for Kinesthetic Notation in High-Dimensional Systems , 2000 .

[6]  Marcelo M. Wanderley,et al.  Trends in Gestural Control of Music , 2000 .

[7]  Marcelo M. Wanderley,et al.  Performer-Instrument Interaction: Applications to Gestural Control of Music , 2001 .

[8]  Perry R. Cook,et al.  Principles for Designing Computer Music Controllers , 2001, NIME.

[9]  Ville Pulkki,et al.  Spatial sound generation and perception by amplitude panning techniques , 2001 .

[10]  Rudolf Rabenstein,et al.  Physical modeling of drums by transfer function methods , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[11]  Tod Machover.  Instruments, Interactivity, and Inevitability , 2002, NIME.

[12]  Nicola Orio,et al.  Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI , 2001, Computer Music Journal.

[13]  Marcelo M. Wanderley,et al.  The Importance of Parameter Mapping in Electronic Instrument Design , 2002, NIME.

[14]  Sergi Jordà,et al.  Digital Instruments and Players: Part I - Efficiency and Apprenticeship , 2004, NIME.

[15]  Perttu Hämäläinen,et al.  Effect of Latency on Playing Accuracy of Two Continuous Sound Instruments Without Tactile Feedback , 2004 .

[16]  Perttu Hämäläinen,et al.  Latency Tolerance for Gesture Controlled Continuous Sound Instrument without Tactile Feedback , 2004, ICMC.

[17]  Matti Karjalainen,et al.  Virtual Air Guitar , 2006 .