A Virtual Reality Platform for Musical Creation: GENESIS-RT

We present GENESIS-RT, a Virtual Reality platform conceived for musical creation. It allows the user to (1) interactively create physically-based musical instruments and sounding objects, and (2) play them in real time in a multisensory fashion, by ways of haptics, 3D visualisation during playing, and sound. The design of this platform aims for full physical coupling, or instrumental interaction, between the musician and the simulated instrument. So doing, it differs from both traditional Digital Musical Instrument architectures and Virtual Reality system architectures. By presenting our environment, we discuss several scientific underlying questions: (1) possible ways to manage simultaneous audio-haptic-visual cooperation during real time multisensory simulations; (2) the Computer Aided Design functionalities for the creation of new physically-based musical instruments and sounding objects, and (3) the synchronous real time features, in terms of software and hardware architecture. Finally the article reviews a series of exemplary models and instrumental situations using the proposed platform.

[1]  John M. Chowning,et al.  The Synthesis of Complex Audio Spectra by Means of Frequency Modulation , 1973 .

[2]  Claude Cadoz,et al.  Interactive Physical Design and Haptic Playing of Virtual Musical Instruments , 2013, ICMC.

[3]  Claude Cadoz,et al.  Sound Synthesis and Musical Composition by Physical Modelling of Self-Sustained Oscillating Structures , 2007 .

[4]  Christophe Vergez,et al.  The BRASS Project, from Physical Models to Virtual Musical Instruments: Playability Issues , 2005, CMMR.

[5]  Martin Hachet,et al.  Piivert: Percussion-based interaction for immersive virtual environments , 2010, 2010 IEEE Symposium on 3D User Interfaces (3DUI).

[6]  Claude Cadoz,et al.  Computational Physics : A Modeler - Simulator for animated physical Objects , 1991, Eurographics.

[7]  S Puckette Miller,et al.  Pure Data : another integrated computer music environment , 1996 .

[8]  Curtis Roads,et al.  Responsive Input Devices and Sound Synthesis by Stimulation of Instrumental Mechanisms: The Cordis System , 1984 .

[9]  Marcelo M. Wanderley,et al.  Towards a Model for Instrumental Mapping in Expert Musical Interaction , 2000, ICMC.

[10]  Claude Cadoz,et al.  A Modular Feedback Keyboard Design , 1990 .

[11]  Claude Cadoz,et al.  Haptics in computer music : a paradigm shift , 2004, ArXiv.

[12]  Marcelo M. Wanderley,et al.  Defining a control standard for easily integrating haptic virtual environments with existing audio/visual systems , 2007, NIME '07.

[13]  Marcelo M. Wanderley,et al.  Audio-Haptic Interaction with the Digital Waveguide Bowed String , 2009, ICMC.

[14]  Marcelo M. Wanderley,et al.  The Importance of Parameter Mapping in Electronic Instrument Design , 2002, NIME.

[15]  Charles Nichols.  The vBow: development of a virtual violin bow haptic human-computer interface , 2002 .

[16]  J. Liu,et al.  Hearing how you touch: Real-time synthesis of contact sounds for multisensory interaction , 2008, 2008 Conference on Human System Interactions.

[17]  Julius O. Smith,et al.  Using Haptics to Assist Performers in Making Gestures to a Musical Instrument , 2009, NIME.

[18]  Julius O. Smith,et al.  Virtual Acoustic Musical Instruments: Review and Update , 2004 .

[19]  Claude Cadoz,et al.  ERGOS: Multi-degrees of Freedom and Versatile Force-Feedback Panoply , 2004 .

[20]  M. Sile O'Modhrain,et al.  Embodied cognition as a motivating perspective for haptic interaction design: A position paper , 2011, 2011 IEEE World Haptics Conference.

[21]  Claude Cadoz,et al.  G³: GENESIS Software Environment Update , 2009, ICMC.

[22]  Sile O'Modhrain,et al.  An enactive approach to the design of new tangible musical instruments , 2006, Organised Sound.

[23]  Lelio Camilleri.  The music machine: Selected readings from “computer music journal” , 2004, Machine Translation.

[24]  Marcelo M. Wanderley,et al.  Gestural control of sound synthesis , 2004, Proceedings of the IEEE.

[25]  Claude Cadoz,et al.  Cordis-anima: A modeling and simulation system for sound and image synthesis , 1993 .

[26]  Marcelo M. Wanderley,et al.  A run-time programmable simulator to enable multi-modal interaction with rigid-body systems , 2009, Interact. Comput..

[27]  Jean-Loup Florens,et al.  Ergotic Sounds: A New Way to Improve Playability, Believability and Presence of Virtual Musical Instruments , 2007 .

[28]  McCartneyJames.  Rethinking the Computer Music Language , 2002 .

[29]  Julius O. Smith,et al.  Physical Modeling Using Digital Waveguides , 1992 .

[30]  Claude Cadoz,et al.  GENESIS: A Friendly Musician-Oriented Environment for Mass-Interaction Physical Modeling , 2002, ICMC.

[31]  C. Cadoz,et al.  Le geste canal de communication homme/machine: la communication "instrumentale" , 1994 .

[32]  Giovanni De Poli,et al.  Representations of musical signals , 1991 .

[33]  James McCartney,et al.  Rethinking the Computer Music Language: SuperCollider , 2002, Computer Music Journal.

[34]  Tellef Kvifte,et al.  On the Description of Mapping Structures , 2008 .

[35]  Claude Cadoz,et al.  Enaction and Enactive Interfaces : A Handbook of terms , 2007 .

[36]  Ross Kirk,et al.  Mapping Strategies for Musical Performance , 2000 .

[37]  Vincent Hayward,et al.  Haptic interfaces and devices , 2004 .

[38]  Tapio Lokki,et al.  Navigation with auditory cues in a virtual environment , 2005, IEEE MultiMedia.

[39]  Brent Gillespie.  The Virtual Piano Action: Design and Implementation , 1994, ICMC.

[40]  Jean-Loup Florens.  Coupleur gestuel interactif pour la commande et le contrôle de sons synthétisés en temps réel. (Interactive gestural coupler for the control of real time synthesized sounds) , 1978 .

[41]  Stephen Travis Pope,et al.  Musical Signal Processing , 1997 .

[42]  Perry R. Cook,et al.  The Synthesis ToolKit (STK) , 1999, ICMC.

[43]  Claude Cadoz,et al.  Organizing Mass-Interaction Physical Models: the CORDIS-ANIMA Musical Instrumentarium , 2009, ICMC.

[44]  Claude Cadoz,et al.  A Goals-Based Review of Physical Modelling , 2005, ICMC.

[45]  Wolfgang Heidrich,et al.  TimbreFields: 3D Interactive Sound Models for Real-Time Audio , 2007, PRESENCE: Teleoperators and Virtual Environments.

[46]  Ross Kirk,et al.  TACTILE FEEDBACK IN THE CONTROL OF A PHYSICAL MODELLING MUSIC SYNTHESISER , 2002 .

[47]  Claude Cadoz.  The Physical Model as Metaphor for Musical Creation: "pico..TERA", a piece generated by physical model , 2002, ICMC.

[48]  Marcelo M. Wanderley,et al.  Trends in Gestural Control of Music , 2000 .

[49]  Julius O. Smith,et al.  Using Haptic Devices to Interface Directly with Digital Waveguide-Based Musical Instruments , 2009, NIME.

[50]  M V Mathews,et al.  The Digital Computer as a Musical Instrument. , 1963, Science.

[51]  Bob L. Sturm,et al.  Proceedings of the International Computer Music Conference , 2011 .

[52]  Vincent Hayward,et al.  Performance Measures for Haptic Interfaces , 1996 .

[53]  Annie Luciani,et al.  Hardware Architecture of a Real-Time Simulator for the CORDIS-ANIMA System: Physical Models, Images, Gestures, Sounds , 1995, Computer Graphics.

[54]  Marcelo M. Wanderley,et al.  Distributed tools for interactive design of heterogeneous signal networks , 2015, Multimedia Tools and Applications.