Music scene description project: Toward audio-based real-time music understanding

Music understanding is an important component of audio‐based interactive music systems. A real‐time music scene description system for the computational modeling of music understanding is proposed. This research is based on the assumption that a listener understands music without deriving musical scores or even fully segregating signals. In keeping with this assumption, our music scene description system produces intuitive descriptions of music, such as the beat structure and the melody and bass lines. Two real‐time subsystems have been developed, a beat‐tracking subsystem and a melody‐and‐bass detection subsystem, which can deal with real‐world monaural audio signals sampled from popular‐music CDs. The beat‐tracking subsystem recognizes a hierarchical beat structure comprising the quarter‐note, half‐note, and measure levels by using three kinds of musical knowledge: of onset times, of chord changes, and of drum patterns. The melody‐and‐bass detection subsystem estimates the F0 (fundamental frequency) of melody and bass lines by using a predominant‐F0 estimation method called PreFEst, which does not rely on the F0’s unreliable frequency component and obtains the most predominant F0 supported by harmonics within an intentionally limited frequency range. Several applications of music understanding are described, including a beat‐driven, real‐time computer graphics and lighting controller.