Challenging conventional assumptions of automated information retrieval with real users: Boolean searching and batch retrieval evaluations

Two common assumptions held by information retrieval researchers are that searching using Boolean operators is inferior to natural language searching and that results from batch-style retrieval evaluations are generalizable to the real-world searching. We challenged these assumptions in the Text Retrieval Conference (TREC) interactive track, with real users following a consensus protocol to search for an instance recall task. Our results showed that Boolean and natural language searching achieved comparable results and that the results from batch evaluations were not comparable to those obtained in experiments with real users.

[1]  Don R. Swanson,et al.  Information Retrieval as a Trial-And-Error Process , 1977, The Library Quarterly.

[2]  Howard R. Turtle.  Natural language vs. Boolean query evaluation: a comparison of retrieval performance , 1994, SIGIR '94.

[3]  Amanda Spink,et al.  Real life information retrieval: a study of user queries on the Web , 1998, SIGF.

[4]  William R. Hersh,et al.  Towards new measures of information retrieval evaluation , 1995, SIGIR '95.

[5]  Chris Buckley,et al.  Pivoted Document Length Normalization , 1996, SIGIR Forum.

[6]  Niels Ole Pors,et al.  Information retrieval, experimental models and statistical analysis , 2000, J. Documentation.

[7]  William R. Hersh,et al.  Relevance and Retrieval Evaluation: Perspectives from Medicine , 1994, J. Am. Soc. Inf. Sci..

[8]  Alistair Moffat,et al.  Exploring the similarity space , 1998, SIGF.

[9]  Stephen E. Robertson,et al.  Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval , 1994, SIGIR '94.

[10]  Bryce Allen.  Cognitive differences in end user searching of a CD-ROM index , 1992, SIGIR '92.

[11]  James Allan,et al.  Aspect windows, 3-D visualizations, and indirect comparisons of information retrieval systems , 1998, SIGIR '98.

[12]  Andrew Turpin,et al.  Do batch and user evaluations give the same results? , 2000, SIGIR '00.

[13]  N Staggers,et al.  Nurse‐Computer Interaction: Staff Performance Outcomes , 1994, Nursing research.

[14]  Ian H. Witten,et al.  Managing Gigabytes: Compressing and Indexing Documents and Images , 1999 .

[15]  Jakob Nielsen,et al.  Measuring usability: preference vs. performance , 1994, CACM.

[16]  William R. Hersh,et al.  An evaluation of interactive Boolean and natural language searching with an online medical textbook , 1995 .

[17]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[18]  Cyril W. Cleverdon,et al.  Factors determining the performance of indexing systems , 1966 .

[19]  G Salton,et al.  Developments in Automatic Text Retrieval , 1991, Science.

[20]  Louis M. Gomez,et al.  Learning to Use a Text Editor: Some Learner Characteristics That Predict Success , 1986, Hum. Comput. Interact..

[21]  Donna Harman,et al.  Overview of the First Text REtrieval Conference. , 1993, SIGIR 1993.

[22]  Chris Buckley,et al.  OHSUMED: an interactive retrieval evaluation and new large test collection for research , 1994, SIGIR '94.

[23]  Amit Singhal,et al.  Pivoted document length normalization , 1996, SIGIR 1996.

[24]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[25]  William R. Hersh,et al.  A task-oriented approach to information retrieval evaluation , 1996 .

[26]  William R. Hersh,et al.  A Large-Scale Comparison of Boolean vs. Natural Language Searching for the TREC-7 Interactive Track , 1998, TREC.

[27]  Paul Over,et al.  TREC-8 interactive track , 1999, SIGF.

[28]  William R. Hersh,et al.  An Evaluation of Interactive Boolean and Natural Language Searching with an Online Medical Textbook , 1995, J. Am. Soc. Inf. Sci..

[29]  Susan T. Dumais,et al.  Iterative Searching in an Online Database , 1991 .

[30]  Michael Keen,et al.  ASLIB CRANFIELD RESEARCH PROJECT FACTORS DETERMINING THE PERFORMANCE OF INDEXING SYSTEMS VOLUME 2 , 1966 .

[31]  Donna Harman,et al.  Information Processing and Management , 2022 .