Some Perspectives on the Evaluation of Information Retrieval Systems

As an introduction to the papers in this special issue, some of the major problems facing investigators evaluating information retrieval systems are presented. These problems include the question of the necessity of using real users, as opposed to subject experts, in making relevance judgements, the possibility of evaluating individual components of the retrieval process, rather than the process as a whole, the kinds of aggregation that are appropriate for the measures used in evaluating information retrieval systems, the value of an analytic or simulatory, as opposed to an experimental, approach in evaluating retrieval systems, the difficulties in evaluating interactive systems, and the kinds of generalization which are possible from information retrieval tests.