MuViSync: Realtime music video alignment

In recent years, the popularity of compressed music files and online music downloads has increased dramatically. Today's users own large digital collections of high quality music on their computers and portable devices to be played in their homes or on the go. In addition, music videos are being offered online both for free and through monthly subscriptions, opening up the opportunity to turn the music listening activity into a multimedia experience. The work presented in this paper addresses the challenge of music audio and music video synchronisation. In particular, we have developed a prototype, named MuViSync, to automatically synchronise music videos to the songs that users are listening to in real-time. At the core of the MuViSync prototype are novel audio synchronization algorithms to tackle the differences in tempo, pitch, sampling rates, structure, and introductions and endings that are common in the various digital recordings of the same song in modern music. The music and the music video are initially aligned and then kept in sync within the limits of human perception. In experiments with 320 matching pairs of audio files and music videos, the proposed algorithms successfully synchronise music and video to within 100 milliseconds of each other in over 90% of the cases.