IMAGECLEF 2004-2005: RESULTS, EXPERIENCES AND NEW IDEAS FOR IMAGE RETRIEVAL EVALUATION

The ImageCLEF image retrieval benchmark was established in 2003 as part of the CLEF (Cross Language Evaluation Forum) to evaluate the retrieval of images from multilingual document collections or retrieval where a query is formulated in a language different from the language of the collection. In 2004, a visual retrieval task was added from the medical domain (using a mixed French/English annotation collection) because the use of visual information has one big advantage: it is inherently language independent. This article describes the achievements of ImageCLEF 2004 by describing its tasks, goals, and the submissions received. The key findings will be explained, which will lead to further ideas to improve retrieval system performance. These will also lead to new ideas for ImageCLEF 2005 that will also be described in this paper, together with the evaluation goals and challenges for ImageCLEF 2005. Systematic performance evaluation is extremely important to show the progress of research in a domain, and there is an important lack of standardised evaluation in image retrieval. ImageCLEF is trying to fill this gap by supplying document collections, image retrieval tasks and topics based on user needs and ground truth for evaluating systems. This creates resources that can subsequently be used to advance research in image retrieval. ImageCLEF also provides a forum in which mixed‐media information retrieval researchers can exchange ideas and technical details through an annual workshop to stimulate discussions and further research. Goal of this article is to motivate research groups from the multimedia retrieval area to participate at ImageCLEF and help propose interesting tasks for further evaluation cam