Measuring search-engine quality and query difficulty: ranking with Target and Freestyle

Instead of using traditional performance measures such as precision and recall, information retrieval performance may be measured by considering the probability that the search engine is optimal and the difficulty associated with retrieving documents with a given query or on a given topic. These measures of desirable characteristics are more easily and more directly interpretable than are traditional measures. The performance of the Target and Freestyle search engines is examined, and is very good. Each query in the CF database is assigned a difficulty number, and these numbers are found to strongly correlate with other measures of retrieval performance such as an E or F value. The query difficulty correlates weakly with query length.

[1]  Robert M. Losee.  Text retrieval and filtering: analytic models of performance , 1998 .

[2]  Robert Burgin.  The Monte Carlo method and the evaluation of retrieval system performance , 1999 .

[3]  Peter Ingwersen,et al.  Cognitive Perspectives of Information Retrieval Interaction: Elements of a Cognitive IR Theory , 1996, J. Documentation.

[4]  Stephen E. Robertson,et al.  On the Evaluation of IR Systems , 1992, Inf. Process. Manag..

[5]  M. E. Maron,et al.  On Relevance, Probabilistic Indexing and Information Retrieval , 1960, JACM.

[6]  C. J. van Rijsbergen,et al.  FOUNDATION OF EVALUATION , 1974 .

[7]  Linda Schamber.  Relevance and Information Behavior. , 1994 .

[8]  Carol L. Barry.  User-defined relevance criteria: an exploratory study , 1994 .

[9]  C. J. van Rijsbergen,et al.  SILOL: A simple logical-linguistic document retrieval system , 1990, Inf. Process. Manag..

[10]  Howard R. Turtle.  Natural language vs. Boolean query evaluation: a comparison of retrieval performance , 1994, SIGIR '94.

[11]  W. M. Shaw.  On the Foundation of Evaluation. , 1986 .

[12]  X. Allan Lu,et al.  Query Expansion/Reduction and its Impact on Retrieval Effectiveness , 1994, TREC.

[13]  Stephen E. Robertson,et al.  Relevance weighting of search terms , 1976, J. Am. Soc. Inf. Sci..

[14]  L. Paris.  Stopping behavior : user persistence in online searching and its relation to optimal cutoff points for information retrieval systems , 1998 .

[15]  B. K. Duval,et al.  Microcomputer applications in the library: exploring Lexis/Nexis: using Microsoft PowerPoint to illustrate FREESTYLETM , 1994 .

[16]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[17]  D. Sperber,et al.  Relevance: Communication and Cognition , 1989 .

[18]  Paul Solomon,et al.  Toward an Understanding of the Dynamics of Relevance Judgment: An Analysis of One Person's Search Behavior , 1998, Inf. Process. Manag..

[19]  W. S. Cooper.  Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems , 1968 .

[20]  Karen Sparck Jones.  A statistical interpretation of term specificity and its application in retrieval , 1972 .

[21]  Andrew Levison.  Ziff-Davis: sale of publishing giant impacts online industry , 1994 .

[22]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[23]  E. Michael Keen,et al.  HOW DOES DIALOG'S TARGET WORK? , 1994 .

[24]  Carol Tenopir,et al.  TARGET and FREESTYLE: DIALOG and Mead join the relevance ranks , 1997 .

[25]  Helen R. Tibbo,et al.  The Cystic Fibrosis Database: Content and Research Opportunities. , 1991 .

[26]  Stephen P. Harter,et al.  Evaluation of information retrieval systems : Approaches, issues, and methods , 1997 .

[27]  Robert Burgin,et al.  Performance Standards and Evaluations in IR Test Collections: Vector-Space and Other Retrieval Models , 1997, Inf. Process. Manag..

[28]  Helen R. Tibbo,et al.  Freestyle vs. Boolean: A Comparison of Partial and Exact Match Retrieval Systems , 1998, Inf. Process. Manag..