The Scientific Evaluation of Music Information Retrieval Systems: Foundations and Future

Computer Music Journal, 28:2, pp. 12–23, Summer 2004 2004 Massachusetts Institute of Technology. Music Information Retrieval (MIR) is a multidisciplinary research endeavor that strives to develop innovative content-based searching schemes, novel interfaces, and evolving networked delivery mechanisms in an effort to make the world’s vast store of music accessible to all. Some teams are developing ‘‘Query-by-Singing’’ and ‘‘Query-by-Humming’’ systems that allow users to interact with their respective music search engines via queries that are sung or hummed into a microphone (e.g., Birmingham et al. 2001; Haus and Pollastri 2001). ‘‘Queryby-Note’’ systems are also being developed wherein searchers construct queries consisting of pitch and/or rhythm information (e.g., Pickens 2000; Doraisamy and Ruger 2002). Input methods for Queryby-Note systems include symbolic interfaces as well as both physical (MIDI) and virtual (Javabased) keyboards. Some teams are working on ‘‘Query-by-Example’’ systems that take prerecorded music in the form of CD or MP3 tracks as their query input (e.g., Haitsma and Kalker 2002; Harb and Chen 2003). The development of comprehensive music recommendation and distribution systems is a growing research area (e.g., Logan 2002; Pauws and Eggen 2002). The automatic generation of playlists for use in personal music systems, based on a wide variety of user-defined criteria, is the goal of this branch of MIR research. Other groups are investigating the creation of music analysis systems to assist those in the musicology and music theory communities (e.g., Barthelemy and Bonardi 2001; Kornstadt 2001). Overviews of MIR’s interdisciplinary research areas can be found in Downie (2003), Byrd and Crawford (2002), and Futrelle and Downie (2002). This article begins with an overview of the current scientific problem facing MIR research. Entitled ‘‘Current Scientific Problem,’’ the opening section also provides a brief explication of the Text Retrieval Conference (TREC) evaluation paradigm that has come to play an important role in the community’s thinking about the testing and evaluation of MIR systems. The sections which follow, entitled ‘‘Data Collection Method’’ and ‘‘Emergent Themes and Commentary,’’ report upon the findings of the Music Information Retrieval (MIR)/ Music Digital Library (MDL) Evaluation Frameworks Project with issues surrounding the creation of a TREC-like evaluation paradigm for MIR as the central focus. ‘‘Building a TREC-Like Test Collection’’ follows next and highlights the progress being made concerning the establishment of the necessary test collections. The ‘‘Summary and Future Research’’ section concludes this article and highlights some of the key challenges uncovered that require further investigation.