A total relevance and document interaction effects model for the evaluation of information retrieval processes

Abstract The article presents a model based on the notion of the total relevance of a set of documents. The concept of a total relevance function is subsequently derived from the notion of cumulated relevance implied in the traditional summation of relevance ratings over the documents in a collection or in retrieved sets of documents. The model is intended to make explicit the perceptual underpinnings of relevance assessments while allowing for the consideration of interdocument dependencies as perceived by the user. Within this framework, it is proposed that an appropriate metric for gauging the performances of information retrieval systems is a measure of the (relative) total relevance that a user can obtain from a set of documents sequentially scanned and evaluated in an information retrieval environment. Some implications of the model are noted.

[1]  M. E. Maron,et al.  An evaluation of retrieval effectiveness for a full-text document-retrieval system , 1985, CACM.

[2]  William B. Rouse,et al.  The value of information and computer-aided information seeking: problem formulation and application to fiction retrieval , 1984, Inf. Process. Manag..

[3]  Gerard Salton,et al.  Research and Development in Information Retrieval , 1982, Lecture Notes in Computer Science.

[4]  Abraham Bookstein,et al.  Information retrieval: A sequential learning process , 1983, J. Am. Soc. Inf. Sci..

[5]  William S. Cooper,et al.  On selecting a measure of retrieval effectiveness , 1973, J. Am. Soc. Inf. Sci..

[6]  David C. Blair.  Searching biases in large interactive document retrieval systems , 1980, J. Am. Soc. Inf. Sci..

[7]  Bertram C. Brookes.  Measurement in information science: Objective and subjective metrical space , 1980, J. Am. Soc. Inf. Sci..

[8]  Peter J. Vigil.  The psychology of online searching , 1983, J. Am. Soc. Inf. Sci..

[9]  Dagobert Soergel,et al.  Is user satisfaction a hobgoblin? , 1976, J. Am. Soc. Inf. Sci..

[10]  Stephen P. Harter,et al.  The Cranfield II Relevance Assessments: A Critical Evaluation , 1971, The Library Quarterly.

[11]  William Goffman,et al.  An indirect method of information retrieval , 1968, Inf. Storage Retr..

[12]  Tefko Saracevic,et al.  RELEVANCE: A review of and a framework for the thinking on the notion in information science , 1997, J. Am. Soc. Inf. Sci..

[13]  William S. Cooper,et al.  A definition of relevance for information retrieval , 1971, Inf. Storage Retr..

[14]  Gerard Salton,et al.  Automatic Information Organization And Retrieval , 1968 .

[15]  William S. Cooper.  The paradoxical role 0f unexamined documents in the evaluation of retrieval effectiveness , 1976, Inf. Process. Manag..

[16]  F. W. Lancaster,et al.  Evaluating the effectiveness of an on-line, natural language retrieval system , 1972, Inf. Storage Retr..

[17]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[18]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[19]  William S. Cooper,et al.  On selecting a measure of retrieval effectiveness part II. Implementation of the philosophy , 1973, J. Am. Soc. Inf. Sci..

[20]  William L. Miller,et al.  THE EXTENSION OF USERS' LITERATURE AWARENESS AS A MEASURE OF RETRIEVAL PERFORMANCE, AND ITS APPLICATION TO MEDLARS , 1971 .

[21]  Stephen E. Robertson.  The probabilistic character of relevance , 1977, Inf. Process. Manag..

[22]  Gerald V. Barrett,et al.  Human Factors Evaluation of a Computer Based Information Storage and Retrieval System , 1968 .

[23]  J. S. Kidd,et al.  REDUNDANCY, RELEVANCE, AND VALUE TO THE USER IN THE OUTPUTS OF INFORMATION RETRIEVAL SYSTEMS , 1976 .

[24]  W. S. Cooper.  Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems , 1968 .

[25]  F. W. Lancaster,et al.  Information retrieval systems; characteristics, testing, and evaluation , 1968 .

[26]  C. W. Cleverdon.  USER EVALUATION OF INFORMATION RETRIEVAL SYSTEMS , 1974 .

[27]  Christine L. Borgman.  End user behavior on an online information retrieval system: a computer monitoring study , 1983, SIGIR 1983.

[28]  David Ellis,et al.  Theory and explanation in information retrieval research , 1984 .