The effectiveness of information retrieval systems: the need for improved explanatory frameworks

Abstract There is a critical lacuna in information retrieval research between the theoretical framework employed in laboratory tests to explain the performance of information retrieval systems, and the sorts of factors which bear on their operational effectiveness. The influence of assumptions as to which factors it is important to take into account in evaluations of the effectiveness of information retrieval systems, derived from the laboratory testing approach in the form of an implicit model of the retrieval situation, has had an inhibiting effect on information retrieval research. Evaluations of the operational effectiveness of information retrieval systems are particularly hampered by the absence of a suitable conceptual framework of explanation. The argument of this paper is that there is a need for more microevaluation of the activities and environments of the users of information systems in order to develop an understanding of the relation of information services to those activities and environments. This sort of understanding should provide more in the way of guidance for the design of more effective systems than is available from current information retrieval theory.