Evaluation Measures for Interactive Information Retrieval

Abstract Several criteria and measures have been proposed and used in evaluating interactive IR performance. There is no agreement about what is a successful IR performance or which are the best existing evaluation measure(s). This study aims to identify the best evaluation measure(s) for interactive IR performance. Twenty measures of IR performance were selected for study in the natural IR environment, involving 40 real end-users from an academic setting with 40 real information problems, interacting with six professional intermediaries searching in large operational IR systems. These end-users were responsible for the costs of their own searches. This study showed that value of search results as a whole is the best single measure of interactive IR performance among the measures selected. Precision, one of the most important traditional measures of effectiveness, is not significantly correlated with success. Users appear to be more concerned with absolute recall than with precision. The study also identified the two more basic factors for future IR evaluation which can account for a much higher proportion of the total variance than value of search results as a whole alone can. Seventeen new success categories were also suggested for future investigation.

[1]  C. Cleverdon.  On the Inverse Relationship of Recall and Precision. , 1972 .

[2]  S. E. Robertson,et al.  THE PARAMETRIC DESCRIPTION OF RETRIEVAL TESTS , 1969 .

[3]  M. E. Maron,et al.  An evaluation of retrieval effectiveness for a full-text document-retrieval system , 1985, CACM.

[4]  Jean Tague-Sutcliffe,et al.  Evaluation of the user interface in an information retrieval system: A model , 1989, Inf. Process. Manag..

[5]  Paul B. Kantor,et al.  A study of information seeking and retrieving. III. Searchers, searches, and overlap , 1988, J. Am. Soc. Inf. Sci..

[6]  Louis M. Gomez,et al.  Formative design evaluation of superbook , 1989, TOIS.

[7]  Paul B. Kantor,et al.  A study of information seeking and retrieving. I. background and methodology , 1988 .

[8]  Ethel Auster,et al.  Search interview techniques and information gain as antecedents of user satisfaction with online bibliographic retrieval , 1984, J. Am. Soc. Inf. Sci..

[9]  Judith A. Tessier.  New Measures of User Satisfaction With Computer Based Literature Searches. , 1977 .

[10]  Eileen E. Hitchingham.  A Study of the Relationship between the Search Interview of the Intermediary Searcher and the Online System User, and the Assessment of Search Results as Judged by the User. Final Report. , 1979 .

[11]  C. W. Cleverdon,et al.  USER EVALUATION OF INFORMATION RETRIEVAL SYSTEMS , 1974 .

[12]  William S. Cooper.  The paradoxical role 0f unexamined documents in the evaluation of retrieval effectiveness , 1976, Inf. Process. Manag..

[13]  R. Tagliacozzo.  Estimating the satisfaction of information users. , 1977, Bulletin of the Medical Library Association.

[14]  Paul B. Kantor,et al.  A study of information seeking and retrieving. I. Background and methodology , 1997, J. Am. Soc. Inf. Sci..

[15]  Louise T. Su.  An investigation to find appropriate measures for evaluating interactive information retrieval , 1991 .

[16]  William S. Cooper,et al.  On selecting a measure of retrieval effectiveness part II. Implementation of the philosophy , 1973, J. Am. Soc. Inf. Sci..

[17]  J A Swets,et al.  Information Retrieval Systems. , 1963, Science.

[18]  Douglas G. Schultz,et al.  A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching. Final Report to the National Science Foundation. Volume II, Appendices. , 1967 .

[19]  William L. Wilkie,et al.  Issues in Marketing's use of Multi-Attribute Attitude Models , 1973 .

[20]  Judith Anne Tessier.  Toward the understanding of user satisfaction: a multiattribute study of user evaluations of computer-based literature searches in medical libraries , 1981 .

[21]  William S. Cooper,et al.  On selecting a measure of retrieval effectiveness , 1973, J. Am. Soc. Inf. Sci..

[22]  Nicholas J. Belkin,et al.  Interaction in information systems : a review of research from document retrieval to knowledge-based systems , 1985 .

[23]  Tefko Saracevic,et al.  RELEVANCE: A review of and a framework for the thinking on the notion in information science , 1997, J. Am. Soc. Inf. Sci..

[24]  Ethel Auster,et al.  A Systems Evaluation of the Educational Information System for Ontario , 1979, J. Am. Soc. Inf. Sci..

[25]  Allen Kent,et al.  Machine literature searching VIII. Operational criteria for designing information retrieval systems , 1955 .

[26]  Pauline A. Atherton,et al.  Presearch Interview Project Progress Report, Covering May 1, 1979 to January 15, 1980. , 1980 .

[27]  Stephen E. Robertson,et al.  THE PARAMETRIC DESCRIPTION OF RETRIEVAL TESTS: PART I: THE BASIC PARAMETERS , 1969 .

[28]  Paul B. Kantor,et al.  A study of information seeking and retrieving. II. Users, questions, and effectiveness , 1988, J. Am. Soc. Inf. Sci..