A novel method for the evaluation of Boolean query effectiveness across a wide operational range

Traditional methods for the system-oriented evaluation of Boolean IR system suffer from validity and reliability problems. Laboratory-based research neglects the searcher and studies suboptimal queries. Research on operational systems fails to make a distinction between searcher performance and system performance. This approach is neither capable of measuring performance at standard points of operation (e.g. across R0.0-R1.0). A new laboratory-based evaluation method for Boolean IR systems is proposed. It is based on a controlled formulation of inclusive query plans, on an automatic conversion of query plans into elementary queries, and on combining elementary queries into optimal queries at standard points of operation. Major results of a large case experiment are reported. The validity, reliability, and efficiency of the method are considered in the light of empirical and analytical test data.

[1]  Stephen P. Harter,et al.  Search term combinations and retrieval overlap: A proposed methodology and case study , 1990, J. Am. Soc. Inf. Sci..

[2]  Nicholas J. Belkin,et al.  Retrieval techniques , 1987 .

[3]  Donna Harman,et al.  The First Text REtrieval Conference (TREC-1) , 1993 .

[4]  P. Willett,et al.  An Introduction to Algorithmic and Cognitive Approaches for Information Retrieval , 1995 .

[5]  Christof N. Schubert,et al.  Information Retrieval Today , 1963 .

[6]  Gerard Salton,et al.  A new comparison between conventional indexing (MEDLARS) and automatic text processing (SMART) , 1972, J. Am. Soc. Inf. Sci..

[7]  M. E. Maron,et al.  An evaluation of retrieval effectiveness for a full-text document-retrieval system , 1985, CACM.

[8]  Paul B. Kantor,et al.  A study of information seeking and retrieving. I. background and methodology , 1988 .

[9]  Stephen P. Harter,et al.  Online Information Retrieval: Concepts, Principles and Techniques , 1986 .

[10]  Eero Sormunen,et al.  A Method for Measuring Wide Range Performance of Boolean Queries in Full-Text Databases , 2000 .

[11]  Timo Niemi,et al.  A deductive data model for query expansion , 1996, SIGIR '96.

[12]  William R. Hersh,et al.  An Evaluation of Interactive Boolean and Natural Language Searching with an Online Medical Textbook , 1995, J. Am. Soc. Inf. Sci..

[13]  Jaana Kekäläinen,et al.  The IR Game - A Tool for Rapid Query Analysis in Cross-Language IR Experiments , 1998 .

[14]  Howard R. Turtle.  Natural language vs. Boolean query evaluation: a comparison of retrieval performance , 1994, SIGIR '94.

[15]  Paolo Toth,et al.  Knapsack Problems: Algorithms and Computer Implementations , 1990 .

[16]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[17]  F. W. Lancaster,et al.  Information retrieval systems; characteristics, testing, and evaluation , 1968 .

[18]  William R. Hersh,et al.  An evaluation of interactive Boolean and natural language searching with an online medical textbook , 1995 .

[19]  Gerard Salton,et al.  Another look at automatic text-retrieval systems , 1986, CACM.

[20]  Tefko Saracevic,et al.  Evaluation of evaluation in information retrieval , 1995, SIGIR '95.

[21]  Joyce A. Mitchell,et al.  The Medline/full-text research project. , 1991 .

[22]  Helen R. Tibbo,et al.  Freestyle vs. Boolean: A Comparison of Partial and Exact Match Retrieval Systems , 1998, Inf. Process. Manag..

[23]  Mirja Iivonen,et al.  Consistency in the Selection of Search Concepts and Search Terms , 1995, Information Processing & Management.

[24]  Allen Newell,et al.  Heuristic programming: ill-structured problems , 1993 .

[25]  Jean Tague-Sutcliffe,et al.  The Pragmatics of Information Retrieval Experimentation Revisited , 1997, Inf. Process. Manag..

[26]  Jacob Shapiro,et al.  Boolean Search: Current State and Perspectives , 1999, J. Am. Soc. Inf. Sci..

[27]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .