Artificial Intelligence in the Concertgebouw

In this paper we present a real-world application (the first of its kind) of machine listening in the context of a live concert in a world-famous concert hall - the Concertgebouw in Amsterdam. A real-time music tracking algorithm listens to the Royal Concertgebouw Orchestra performing Richard Strauss' Alpensinfonie and follows the progress in the sheet music, i.e., continuously tracks the most likely position of the live music in the printed score. This information, in turn, is used to enrich the concert experience for members of the audience by streaming synchronised visual content (the sheet music, explanatory text and videos) onto tablet computers in the concert hall. The main focus of this paper is on the challenges involved in tracking live orchestral music, i.e., how to deal with heavily polyphonic music, how to prepare the data needed, and how to achieve the necessary robustness and precision.

[1]  Arshia Cont,et al.  A Coupled Duration-Focused Architecture for Real-Time Music-to-Score Alignment , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Gaël Richard,et al.  A comparative study of tonal acoustic features for a symbolic level music-to-score alignment , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[3]  Gerhard Widmer,et al.  Adaptive distance normalization for real-time music tracking , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[4]  H. Damasio,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence: Special Issue on Perceptual Organization in Computer Vision , 1998 .

[5]  Gerhard Widmer,et al.  Automatic Page Turning for Musicians via Real-Time Machine Listening , 2008, ECAI.

[6]  Peter Grosche,et al.  High resolution audio synchronization using chroma onset features , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[7]  Christopher Raphael,et al.  Music Plus One and Machine Learning , 2010, ICML.

[8]  A. Arzt.  SIMPLE TEMPO MODELS FOR REAL-TIME MUSIC TRACKING , 2010 .

[9]  Simon Dixon,et al.  An On-Line Time Warping Algorithm for Tracking Musical Performances , 2005, IJCAI.

[10]  Jordi Janer,et al.  Audio-to-score Alignment at the Note Level for Orchestral Recordings , 2014, ISMIR.

[11]  Alin Achim,et al.  2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO 2012) , 2013 .

[12]  Florian Krebs,et al.  Tracking rests and Tempo changes: Improved Score following with Particle filters , 2013, ICMC.

[13]  Roger B. Dannenberg,et al.  An On-Line Algorithm for Real-Time Accompaniment , 1984, ICMC.

[14]  Gerhard Widmer,et al.  Automatic Alignment of Music Performances with Structural Differences , 2013, ISMIR.

[15]  George Tzanetakis,et al.  Polyphonic audio matching and alignment for music retrieval , 2003, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684).

[16]  Simon Dixon,et al.  Robust Joint Alignment of Multiple Versions of a Piece of Music , 2016, ISMIR.

[17]  Barry Vercoe,et al.  The Synthetic Performer in The Context of Live Performance , 1984, International Conference on Mathematics and Computing.