VRAPS (Visual Rhythm-based Audio Playback System) is an interactive multimedia application that uses a novel visual rhythm detection technique to allow a user to control the playback speed of an audio signal by moving in front of a video camera. As traditionally defined in the context of music, a beat represents a distinctive musical event such as the hitting of a drum or the start of a new melodic note. Similarly, we define a visual beat to indicate distinctive visual events such as rhythmic gestures and dance motions. In this paper, we propose a method for automatically detecting visual beats and tempo from a video signal. As a demonstration of this technique, we present an interactive, real-time audio playback system that continuously adjusts the playback speed of an audio signal based on the visual rhythm detected from a real-time video capture device.
[1]
Alexander Refsum Jensenius,et al.
Body Movement in Music Information Retrieval
,
2009,
ISMIR.
[2]
J.-Y. Bouguet,et al.
Pyramidal implementation of the lucas kanade feature tracker
,
1999
.
[3]
Mark B. Sandler,et al.
A tutorial on onset detection in music signals
,
2005,
IEEE Transactions on Speech and Audio Processing.
[4]
Qi Tian,et al.
Foreground object detection from videos containing complex background
,
2003,
MULTIMEDIA '03.
[5]
Alexander Refsum Jensenius.
Using Motiongrams in the Study of Musical Gestures
,
2006,
ICMC.
[6]
Simon Dixon,et al.
A Review of Automatic Rhythm Description Systems
,
2005,
Computer Music Journal.