Exploring Virtual Sound Environments with Mobile Devices

The aim of this research is to explore virtual sound environments with mobile devices, using iOS as a main platform and Pure Data (Pd) as a backend for sound processing. The underlying calculations are based on human’s Interaural Time Difference (ITD) and distance between virtual sound sources. As a result, the developed application allows users to “walk” around virtual concert, as well as to experiment with positions of sound sources by moving them manually with a GUI. 1) Introduction: Positioning musical instruments in 3D spaces has been always a topic of interest for musicians and sound engineers, since the relative position of a listener and a sound source affects the perceptual characteristics of the audition experience. Thus, the ability to effortlessly arrange sound environments by virtually re-arranging them is desirable, especially if combined with the options provided by mobile devices. The motivating idea of this project is to allow users to “walk” around several virtual instruments in a shared virtual space, and hearing how their sound mix changes in real-time. Alternatively, users can remain static and modify positions of the sound sources. Fig. 1. Application dataflow 2) Implementation: The project consists of two parts: a Pd module and iOS application, as shown in Fig. 1. The Pd module comprises three stages: • Sound loading: This routine loads recorded sound files and plays them continuously while the main application is running. • Positioning: This routine retrieves rectangular coordinates of the virtual listener (sink) and instruments (sources) from the GUI and passes them to the sound processing routine. • Sound processing: This is the core of the system, which creates the spatialization illusion. It takes the current positions of the listener and the sound sources and compiles the soundscape by computing an audio mix. For spatialization, the system relies on the relative position of sources and listener. Position is determined by their actual locations whereas orientation is given by the virtual rotation of the listener’s head. The latter is used to determine approximate Interaural Time Difference (ITD) based on sphere-shaped human head [2] and distance-based amplitude value of a virtual sound source. ITD is defined as