A Task-Oriented Approach for the Development of a Test Collection for Music Information Retrieval

This paper addresses the design of a test collection for CBMR, by applying a task-oriented approach. After having created a test document set, it is proposed to provide assessors with a number of topics and to build musical queries starting from music documents, or documents excerpts, being particularly relevant to the topic. Relevance judgments are then built based on the similarity between this document and the other documents in the collection. Being far to solve all the problematic issues related to the design of a test collection, this paper aims to give some insights and procedures to carry on the test collection design process, which requires many human resources and different types of expertise.