Anticipatory Networked Communications for Live Musical Interactions of Acoustic Instruments

Performing music over computer networks requires highly reliable infrastructures with respect to latency elimination, bandwidth availability and error alleviation. Hence, commonly available networks or large communication distance cannot effectively cope with real-time, high-fidelity and uninterrupted musical communications. An intuitive solution to this problem can be offered by employing mechanisms for musical prediction and anticipation which can help mitigate technical impediments. The idea is that knowing in advance the evolution of a music piece can offer effective reduction of communication latencies and data rates. Furthermore, musical anticipation can be used to eliminate errors owing to data loss in the network transmission path. We present the development of a software prototype that uses anticipatory processes to support networked music performance. The adopted methodology is inspired by computer accompaniment systems, in which a software agent follows the performance of a human musician in real-time, by synchronizing a pre-recorded musical accompaniment to the expressive nuances of live interpretation. The paper presents an overview of relevant research initiatives, the computational approach implemented for anticipatory networked performances and an experimental validation that demonstrates current progress and reveals constraints and challenges pending further investigation.

[1]  Barry Vercoe,et al.  The Synthetic Performer in The Context of Live Performance , 1984, International Conference on Mathematics and Computing.

[2]  Roger B. Dannenberg,et al.  Real-Time Computer Accompaniment of Keyboard Performances , 1985, ICMC.

[3]  D. Schwarz,et al.  Corpus-Based Concatenative Synthesis , 2007, IEEE Signal Processing Magazine.

[4]  Xiaotao Wu,et al.  Enhancing Application-Layer Multicast for P2P Conferencing , 2007, 2007 4th IEEE Consumer Communications and Networking Conference.

[5]  Christopher Raphael,et al.  Orchestral Musical Accompaniment from Synthesized Audio , 2003, ICMC.

[6]  Roger B. Dannenberg,et al.  Methods and Prospects for Human–Computer Performance of Popular Music , 2014, Computer Music Journal.

[7]  Christopher Raphael,et al.  Music Plus One: A System for Expressive and Flexible Musical Accompaniment , 2001, ICMC.

[8]  A. Wilgus,et al.  High quality time-scale modification for speech , 1985, ICASSP '85. IEEE International Conference on Acoustics, Speech, and Signal Processing.

[9]  Yoshitaka Nakajima,et al.  Auditory Scene Analysis: The Perceptual Organization of Sound Albert S. Bregman , 1992 .

[10]  Nicola Orio,et al.  Alignment of Monophonic and Polyphonic Music to a Score , 2001, ICMC.

[11]  M. Davies,et al.  A HYBRID APPROACH TO MUSICAL NOTE ONSET DETECTION , 2002 .

[12]  Pedro Rebelo,et al.  Networked Music Performance: State of the Art , 2007 .

[13]  Chris Chafe,et al.  Effect of Time Delay on Ensemble Accuracy , 2004 .

[14]  A. A. Sawchuk,et al.  From remote media immersion to Distributed Immersive Performance , 2003, ETP '03.

[15]  Eric Larson,et al.  Real-Time Time-Domain Pitch Tracking Using Wavelets , 2006 .

[16]  Elaine Chew,et al.  Towards Seamless Network Music Performance: Predicting an Ensemble's Expressive Decisions for Distributed Performance , 2014, ISMIR.

[17]  Tetsuya Ogata,et al.  Real-Time Audio-to-Score Alignment Using Particle Filter for Coplayer Music Robots , 2011, EURASIP J. Adv. Signal Process..

[18]  Özgür Izmirli,et al.  Audio-to-audio Alignment using Particle filters to Handle Small and Large Scale Performance Discrepancies , 2012, ICMC.

[19]  Barry Vercoe,et al.  Recognition and prediction in a network music performance system for Indian percussion , 2007, NIME '07.

[20]  Jeff Pressing,et al.  Improvisation: Methods and models. , 1988 .

[21]  Chris Chafe,et al.  JackTrip: Under the Hood of an Engine for Network Audio , 2010, ICMC.

[22]  Peter F. Driessen,et al.  The Effects of Network Delay on Tempo in Musical Performance , 2011, Computer Music Journal.

[23]  Yoichi Muraoka,et al.  RMCP: Remote Music Control Protocol - Design and Applications , 1997, ICMC.

[24]  Roger B. Dannenberg,et al.  An On-Line Algorithm for Real-Time Accompaniment , 1984, ICMC.

[25]  Timothy B. Terriberry,et al.  High-Quality, Low-Delay Music Coding in the Opus Codec , 2016, ArXiv.

[26]  lvaro Barbosa,et al.  Displaced Soundscapes: A Survey of Network Systems for Music and Sonic Art Creation , 2003, Leonardo Music Journal.

[27]  William F. Walker,et al.  A computer participant in musical improvisation , 1997, CHI.

[28]  Udo Zölzer,et al.  High Quality Time-Domain Pitch Shifting Using PSOLA and Transient Preservation , 2010 .

[29]  Alain Renaud,et al.  Playing the Network: the Use of Time delays as Musical Devices , 2008, ICMC.

[30]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[31]  E. Clarke.  Generative principles in music performance. , 1988 .

[32]  Robert Rowe,et al.  Machine Musicianship , 2001 .

[33]  J. Sloboda.  Generative Processes in Music: The Psychology of Performance, Improvisation, and Composition , 1985 .

[34]  Jeremy R. Cooperstock,et al.  Real-Time Streaming of Multichannel Audio Data over Internet , 2000 .

[35]  Tetsuya Ogata,et al.  Thereminist robot: Development of a robot theremin player with feedforward and feedback arm control based on a Theremin's pitch model , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[36]  George Tzanetakis,et al.  Polyphonic audio matching and alignment for music retrieval , 2003, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684).

[37]  E. Lindemann.  Music Synthesis with Reconstructive Phrase Modeling , 2007, IEEE Signal Processing Magazine.

[38]  S. Dixon.  ONSET DETECTION REVISITED , 2006 .

[39]  David Mendonça,et al.  Cognition in Jazz Improvisation: An Exploratory Study , 2004 .

[40]  Jeremy R. Cooperstock,et al.  The recording studio that spanned a continent , 2001, Proceedings First International Conference on WEB Delivering of Music. WEDELMUSIC 2001.

[41]  Demosthenes Akoumianakis,et al.  Exploring New Perspectives in Network Music Performance: The DIAMOUSES Framework , 2010, Computer Music Journal.

[42]  Golo Föllmer.  Electronic, aesthetic and social factors in Net music , 2005 .

[43]  Roger B. Dannenberg.  Human Computer Music Performance , 2012, Multimodal Music Processing.

[44]  Ajay Kapur,et al.  Interactive Network Performance: a dream worth dreaming? , 2005, Organised Sound.

[45]  Edward W. Large,et al.  Perceiving temporal regularity in music , 2002, Cogn. Sci..

[46]  Arshia Cont,et al.  A unified approach to real time audio-to-score and audio-to-audio alignment using sequential Montecarlo inference techniques , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[47]  Arshia Cont,et al.  Improvement of Observation Modeling for Score Following , 2004 .

[48]  Roger B. Dannenberg,et al.  Real-time scheduling and computer accompaniment , 1989 .

[49]  Perttu Hämäläinen,et al.  EFFECT OF LATENCY ON PLAYING ACCURACY OF TWO GESTURE CONTROLLED CONTINUOUS SOUND INSTRUMENTS WITHOUT TACTILE FEEDBACK , 2004 .

[50]  Hideki Kawahara,et al.  YIN, a fundamental frequency estimator for speech and music. , 2002, The Journal of the Acoustical Society of America.

[51]  Max Mühlhäuser,et al.  Parameter Controlled Remote Performance (Pcrp): Playing Together Despite High delay , 2008, ICMC.

[52]  John Bischoff,et al.  Music for an Interactive Network of Microcomputers , 1978 .

[53]  Matthew Wright,et al.  Open SoundControl: A New Protocol for Communicating with Sound Synthesizers , 1997, ICMC.

[54]  Georg Hajdu,et al.  On the Evolution of Music Notation in Network Music Environments , 2009 .

[55]  Michael Young.  Identity and Intimacy in Human-Computer Improvisation , 2010, Leonardo Music Journal.