Interactive virtual percussion instruments on mobile devices

We present a multimodal virtual percussion instrument system on consumer mobile devices that allows users to design and configure customizable virtual percussion instruments and interact with them in real time. Users can create virtual instruments of different materials and shapes interactively, by editing and selecting the desired characteristics. Both the visual and auditory feedback are then computed on the fly to automatically correspond to the instrument properties and user interaction. We utilize efficient 3D input processing algorithms to approximate and represent real-time multi-touch input with key meta properties and adopt fast physical modeling to synthesize sounds. Despite the relatively limited computing resources on mobile devices, we are able to achieve rich and responsive multimodal feedback based on real-time user input. A pilot study is conducted to assess the effectiveness of the system.

[1]  Julius O. Smith,et al.  Performance Expression in Commuted Waveguide Synthesis of Bowed Strings , 1995, ICMC.

[2]  Ming C. Lin,et al.  Auditory Perception of Geometry-Invariant Material Properties , 2013, IEEE Transactions on Visualization and Computer Graphics.

[3]  A. Shabana.  Vibration of Discrete and Continuous Systems , 1996, Mechanical Engineering Series.

[4]  Stefania Serafin,et al.  Playability Evaluation of a Virtual Bowed String Instrument , 2003, NIME.

[5]  Perry R. Cook,et al.  Banded Waveguides: Towards Physical Modeling of Bowed Bar Percussion Instruments , 1999, ICMC.

[6]  Ming C. Lin,et al.  Example-guided physically based modal sound synthesis , 2013, ACM Trans. Graph..

[7]  Ming C. Lin,et al.  Synthesizing contact sounds between textured models , 2010, 2010 IEEE Virtual Reality Conference (VR).

[8]  Victor Lazzarini.  Erratum: New Digital Musical Instruments: Control and Interaction Beyond the Keyboard , 2008, Computer Music Journal.

[9]  Perry R. Cook,et al.  Real Sound Synthesis for Interactive Applications , 2002 .

[10]  Ming C. Lin,et al.  Tabletop Ensemble: touch-enabled virtual percussion instruments , 2012, I3D '12.

[11]  Marc Battier.  Representations of Musical Signals ed. by Giovanni de Poli, Aldo Piccialli and Curtis Roads (review) , 1992 .

[12]  Ming C. Lin,et al.  Integrated multimodal interaction using normal maps , 2015, Graphics Interface.

[13]  Marcelo M. Wanderley,et al.  New Digital Musical Instruments: Control And Interaction Beyond the Keyboard (Computer Music and Digital Audio Series) , 2006 .

[14]  M. Sile O'Modhrain,et al.  Physical models and musical controllers: designing a novel electronic percussion instrument , 2007, NIME '07.

[15]  Gil Weinberg,et al.  The interactive robotic percussionist - new developments in form, mechanics, perception and interaction design , 2007, 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI).