LyricAlly: automatic synchronization of acoustic musical signals and textual lyrics

We present a prototype that automatically aligns acoustic musical signals with their corresponding textual lyrics, in a manner similar to manually-aligned karaoke. We tackle this problem using a multimodal approach, where the appropriate pairing of audio and text processing helps create a more accurate system. Our audio processing technique uses a combination of top-down and bottom-up approaches, combining the strength of low-level audio features and high-level musical knowledge to determine the hierarchical rhythm structure, singing voice and chorus sections in the musical audio. Text processing is also employed to approximate the length of the sung passages using the textual lyrics. Results show an average error of less than one bar for per-line alignment of the lyrics on a test bed of 20 songs (sampled from CD audio and carefully selected for variety). We perform holistic and per-component testing and analysis and outline steps for further development.

[1]  Masataka Goto,et al.  An Audio-based Real-time Beat Tracking System for Music With or Without Drum-sounds , 2001 .

[2]  M. Goto.  A chorus-section detecting method for musical audio signals , 2003, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684).

[3]  Daniel P. W. Ellis,et al.  Locating singing voice segments within music signals , 2001, Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575).

[4]  M. Furini,et al.  Audio-text synchronization inside mp3 files: a new approach and its implementation , 2004, First IEEE Consumer Communications and Networking Conference, 2004. CCNC 2004..

[5]  Ye Wang,et al.  Singing voice detection in popular music , 2004, MULTIMEDIA '04.

[6]  Daniel P. W. Ellis,et al.  Ground-truth transcriptions of real music from force-aligned MIDI syntheses , 2003, ISMIR.

[7]  Say Wei Foo,et al.  Stress Classification Using Subband Based Features , 2003 .

[8]  George Tzanetakis,et al.  Song-specific bootstrapping of singing voice structure , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[9]  Ning Hu,et al.  Polyphonic Audio Matching for Score Following and Intelligent Audio Editors , 2003, ICMC.

[10]  Ren-Yuan Lyu,et al.  An automatic singing transcription system with multilingual singing lyric recognizer and robust melody tracker , 2003, INTERSPEECH.

[11]  Daniel P. W. Ellis,et al.  USING VOICE SEGMENTS TO IMPROVE ARTIST CLASSIFICATION OF MUSIC , 2002 .

[12]  Ye Wang,et al.  Key determination of acoustic musical signals , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[13]  Jon W. Dunn,et al.  A digital library data model for music , 2002, JCDL '02.

[14]  Meinard Müller,et al.  Automatic synchronization of music data in score-, MIDI- and PCM-format , 2003, ISMIR.