Virtual Reality Musical Instruments: State of the Art, Design Principles, and Future Directions

The rapid development and availability of low-cost technologies have created a wide interest in virtual reality. In the field of computer music, the term “virtual musical instruments” has been used for a long time to describe software simulations, extensions of existing musical instruments, and ways to control them with new interfaces for musical expression. Virtual reality musical instruments (VRMIs) that include a simulated visual component delivered via a head-mounted display or other forms of immersive visualization have not yet received much attention. In this article, we present a field overview of VRMIs from the viewpoint of the performer. We propose nine design guidelines, describe evaluation methods, analyze case studies, and consider future challenges.

[1]  Ivan E. Sutherland,et al.  The Ultimate Display , 1965 .

[2]  Julius O. Smith,et al.  Extensions of the Karplus-Strong Plucked-String Algorithm , 1983 .

[3]  T. Stoffregen,et al.  An ecological Theory of Motion Sickness and Postural Instability , 1991 .

[4]  Carolina Cruz-Neira,et al.  Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE , 2023 .

[5]  Claude Cadoz,et al.  Cordis-anima: A modeling and simulation system for sound and image synthesis , 1993 .

[6]  Durand R. Begault,et al.  3-D Sound for Virtual Reality and Multimedia Cambridge , 1994 .

[7]  Vesa Välimäki,et al.  Virtual musical instruments — natural sound using physical models , 1996, Organised Sound.

[8]  Ivan Poupyrev,et al.  The go-go interaction technique: non-linear mapping for direct manipulation in VR , 1996, UIST '96.

[9]  Matthew Lombard,et al.  At the Heart of It All: The Concept of Presence , 2006 .

[10]  Axel G. E. Mulder.  Design of virtual three-dimensional instruments for sound control , 1998 .

[11]  Pierre Lévy,et al.  Becoming Virtual: Reality in the Digital Age , 1998 .

[12]  Philip Zhai.  Get Real: A Philosophical Adventure in Virtual Reality , 1998 .

[13]  Steven van de Par,et al.  Auditory-visual interaction: from fundamental research in cognitive psychology to (possible) applications , 1999, Electronic Imaging.

[14]  Mel Slater,et al.  Public Speaking in Virtual Reality: Facing an Audience of Avatars , 1999, IEEE Computer Graphics and Applications.

[15]  S. Gallagher.  Philosophical conceptions of the self: implications for cognitive science , 2000, Trends in Cognitive Sciences.

[16]  Julius O. Smith,et al.  Qualitative and Quantitive Assessment of a Virtual Bowed String Instrument , 2000, ICMC.

[17]  Golan Levin.  Painterly interfaces for audiovisual performance , 2000 .

[18]  Joseph J. LaViola,et al.  A discussion of cybersickness in virtual environments , 2000, SGCH.

[19]  Chris Chafe,et al.  Playing by feel: incorporating haptic feedback into computer-based musical instruments , 2001 .

[20]  Perry R. Cook,et al.  Principles for Designing Computer Music Controllers , 2001, NIME.

[21]  F. Brooks,et al.  Physiological Reaction as an Objective Measure of Presence in Virtual Environments , 2001 .

[22]  Matthew Wright,et al.  Problems and prospects for intimate musical control of computers , 2001 .

[23]  Nicola Orio,et al.  Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI , 2001, Computer Music Journal.

[24]  Perry R. Cook,et al.  Real Sound Synthesis for Interactive Applications , 2002 .

[25]  Sergi Jordà,et al.  INTERACTIVE MUSIC SYSTEMS FOR EVERYONE: EXPLORING VISUAL FEEDBACK AS A WAY FOR CREATING MORE INTUITIVE, EFFICIENT AND LEARNABLE INSTRUMENTS , 2003 .

[26]  Jo Smewing.  Exploring new dimensions in texture analysis , 2003 .

[27]  Frank Biocca,et al.  The Effect of the Agency and Anthropomorphism on Users' Sense of Telepresence, Copresence, and Social Presence in Virtual Environments , 2003, Presence: Teleoperators & Virtual Environments.

[28]  S. Jordà.  Instruments and Players: Some Thoughts on Digital Lutherie , 2004 .

[29]  Teemu Mäki-Patola,et al.  Physics-based modeling of musical instruments for interactive virtual reality , 2004, IEEE 6th Workshop on Multimedia Signal Processing, 2004..

[30]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[31]  Perttu Hämäläinen,et al.  EFFECT OF LATENCY ON PLAYING ACCURACY OF TWO GESTURE CONTROLLED CONTINUOUS SOUND INSTRUMENTS WITHOUT TACTILE FEEDBACK , 2004 .

[32]  Tapio Takala,et al.  Experiments with Virtual Reality Instruments , 2005, NIME.

[33]  C. Chafe.  Oxygen flute: A computer music instrument that grows , 2005 .

[34]  Xavier Rodet,et al.  Study of haptic and visual interaction for sound and music control in the phase project , 2005 .

[35]  Gerardo Herrera,et al.  Agency and Presence: A Common Dependence on Subjectivity? , 2006, PRESENCE: Teleoperators and Virtual Environments.

[36]  Matti Karjalainen,et al.  Virtual Air Guitar , 2006 .

[37]  Serafín.  Aalborg Universitet Synthesis and control of everyday sounds reconstructing Russolo ’ s , 2006 .

[38]  Stefania Serafin,et al.  Synthesis and control of everyday sound reconstructing Russolo's Intonarumori , 2006, NIME.

[39]  Susanne Bødker,et al.  When second wave HCI meets third wave challenges , 2006, NordiCHI '06.

[40]  Will Kurt.  An Interview with Jaron Lanier , 2007 .

[41]  Stefania Serafin,et al.  Cyberinstruments via Physical Modeling Synthesis: Compositional Applications , 2007, Leonardo Music Journal.

[42]  Stefania Serafin,et al.  Virtual Reality Instruments capable of changing Dimensions in Real-time , 2007 .

[43]  Linda Candy,et al.  Designing and evaluating virtual musical instruments: facilitating conversational user interaction , 2008 .

[44]  G. Paine.  Towards Unified Design Guidelines for New Interfaces for Musical Expression , 2009, Organised Sound.

[45]  Robert Hamilton.  Building Interactive Networked Musical Environments Using q3osc , 2009 .

[46]  Perry R. Cook,et al.  Re-Designing Principles for Computer Music Controllers: a Case Study of SqueezeVox Maggie , 2009, NIME.

[47]  Mel Slater,et al.  Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[48]  Andrew Johnston,et al.  Interfaces for musical expression based on simulated physical models , 2009 .

[49]  Smith,et al.  Physical audio signal processing : for virtual musical instruments and audio effects , 2010 .

[50]  Ben Shneiderman.  Interview of Ben Shneiderman , 2010 .

[51]  Charles Ford.  Musical Presence: Towards a New Philosophy of Music , 2010 .

[52]  Andrew Johnston.  Beyond Evaluation: Linking Practice and Theory in New Musical Interface Design , 2011, NIME.

[53]  Marcelo M. Wanderley,et al.  Examining the Effects of Embedded Vibrotactile Feedback on the Feel of a Digital Musical Instrument , 2011, NIME.

[54]  M. Sile O'Modhrain,et al.  A Framework for the Evaluation of Digital Musical Instruments , 2011, Computer Music Journal.

[55]  Cumhur Erkut,et al.  A Structured Design and Evaluation Model with Application to Rhythmic Interaction Displays , 2011, NIME.

[56]  Martin Hachet,et al.  Interacting with 3D Reactive Widgets for Musical Performance , 2011 .

[57]  Cumhur Erkut,et al.  Design and Evaluation of Human–Computer Rhythmic Interaction in a Tutoring System , 2011, Computer Music Journal.

[58]  S. Bouchard,et al.  NEW DIMENSIONS IN THE ASSESSMENT OF VIRTUAL REALITY INDUCED SIDE EFFECTS EXPLORING NEW DIMENSIONS IN THE ASSESSMENT OF VIRTUAL REALITY INDUCED SIDE EFFECTS , 2012 .

[59]  Sriram Subramanian,et al.  Rouages: Revealing the Mechanisms of Digital Musical Instruments to the Audience , 2013, NIME.

[60]  I. Poupyrev,et al.  AIREAL: interactive tactile experiences in free air , 2013, ACM Trans. Graph..

[61]  Juraj Kojs.  Virtual reality and sonic cyberspaces: augmentation, hybridisation and abstraction , 2013, Int. J. Arts Technol..

[62]  Claude Cadoz,et al.  A Virtual Reality Platform for Musical Creation: GENESIS-RT , 2013, CMMR.

[63]  Mel Slater,et al.  Drumming in Immersive Virtual Reality: The Body Shapes the Way We Play , 2013, IEEE Transactions on Visualization and Computer Graphics.

[64]  Basheer Tome,et al.  Andante: Walking Figures on the Piano Keyboard to Visualize Musical Motion , 2014, NIME.

[65]  Simon Davis,et al.  A Systematic Review of Cybersickness , 2014, IE.

[66]  Ge Wang,et al.  Principles of Visual Design for Computer Music , 2014, ICMC.

[67]  Florent Berthaut,et al.  Scenography of immersive virtual musical instruments , 2014, 2014 IEEE VR Workshop: Sonic Interaction in Virtual Environments (SIVE).

[68]  Antonella De Angeli,et al.  Musical Interface Design: An Experience-oriented Framework , 2014, NIME.

[69]  Susanne Bødker,et al.  Third-wave HCI, 10 years later---participation and sharing , 2015, Interactions.

[70]  Cumhur Erkut,et al.  The digital Intonarumori , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[71]  Dominik Rausch,et al.  Cirque des bouteilles: The art of blowing on bottles , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[72]  Marcelo Knörich Zuffo,et al.  Crosscale: A 3D virtual musical instrument interface , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[73]  Ryan P. McMahan,et al.  Wedge: A musical interface for building and playing composition-appropriate immersive environments , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[74]  John Fillwalk.  ChromaChord: A virtual musical instrument , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[75]  Ferran Argelaguet,et al.  The role of interaction in virtual embodiment: Effects of the virtual hand representation , 2016, 2016 IEEE Virtual Reality (VR).

[76]  Steven K. Feiner,et al.  Combating VR sickness through subtle dynamic field-of-view modification , 2016, 2016 IEEE Symposium on 3D User Interfaces (3DUI).