Parallel online time warping for real-time audio-to-score alignment in multi-core systems

The audio-to-score framework consists of two separate stages: preprocessing and alignment. The alignment is commonly solved through offline dynamic time warping (DTW), which is a method to find the path over the distortion matrix with the minimum cost to determine the relation between the performance and the musical score times. In this work we propose a parallel online DTW solution based on a client–server architecture. The current version of the application has been implemented for multi-core architectures ($$\times $$×86, $$\times $$×64 and ARM), thus covering either powerful systems or mobile devices. An extensive experimentation has been conducted to validate the software. The experiments also show that our framework allows to achieve a good score alignment within the real-time window using parallel computing techniques.

[1]  Xavier Rodet,et al.  Globally Optimal Short-Time Dynamic Time Warping, Application to Score to Audio Alignment , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[2]  Meinard Müller,et al.  An Efficient Multiscale Approach to Audio Synchronization , 2006, ISMIR.

[3]  F. Itakura,et al.  Minimum prediction residual principle applied to speech recognition , 1975 .

[4]  Christopher Raphael,et al.  Aligning music audio with symbolic scores using a hybrid graphical model , 2006, Machine Learning.

[5]  Gerhard Widmer,et al.  Towards Effective 'Any-Time' Music Tracking , 2010, STAIRS.

[6]  Ian H. Witten,et al.  Towards the digital music library: tune retrieval from acoustic input , 1996, DL '96.

[7]  Meinard Müller,et al.  Towards an Efficient Algorithm for Automatic Score-to-Audio Synchronization , 2004, ISMIR.

[8]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.

[9]  Ning Hu,et al.  Polyphonic Audio Matching for Score Following and Intelligent Audio Editors , 2003, ICMC.

[10]  Roger B. Dannenberg,et al.  An Intelligent Multi-Track audio Editor , 2007, ICMC.

[11]  Nicola Orio,et al.  Alignment of Monophonic and Polyphonic Music to a Score , 2001, ICMC.

[12]  Bryan Pardo,et al.  Online Score-Informed Source Separation with Adaptive Instrument Models , 2015 .

[13]  Meinard Müller,et al.  Handling Repeats and Jumps in Score-performance Synchronization , 2010, ISMIR.

[14]  C. Joder,et al.  A Conditional Random Field Framework for Robust and Scalable Audio-to-Score Matching , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[15]  Nicolás Ruiz-Reyes,et al.  An Audio to Score Alignment Framework Using Spectral Factorization and Dynamic Time Warping , 2015, ISMIR.

[16]  Nicolás Ruiz-Reyes,et al.  Constrained non-negative sparse coding using learnt instrument templates for realtime music transcription , 2013, Eng. Appl. Artif. Intell..

[17]  Gerhard Widmer,et al.  Automatic Page Turning for Musicians via Real-Time Machine Listening , 2008, ECAI.

[18]  Bryan Pardo,et al.  Soundprism: An Online System for Score-Informed Source Separation of Music Audio , 2011, IEEE Journal of Selected Topics in Signal Processing.

[19]  Dan Morris,et al.  MySong: automatic accompaniment generation for vocal melodies , 2008, CHI.

[20]  Simon Dixon,et al.  LIVE TRACKING OF MUSICAL PERFORMANCES USING ON-LINE TIME WARPING , 2005 .