On the Evaluation of Snippet Selection for Information Retrieval

In this paper we take a critical look at the evaluation method of WebCLEF 2007. The suitability of the evaluation method can be seen from two sides, namely from a participating system and a non participating system. A participant has the advantage that the evaluation is partly based upon his output. In this paper we will investigate if the size of the pool of snippets, the implementation of the evaluation method and the quality of the assessments is sufficient enough for reliable evaluation. Unfortunately we have to conclude that the evaluation is not suitable. Therefore some alternative evaluation methods will be discussed concluding in a recommendation to improve the evaluation of WebCLEF.