A Retrospective Evaluation Method for Exact-Match and Best-Match Queries Applying an Interactive Query Performance Analyser

A retrospective method for the performance comparison of Boolean and best-match queries is introduced. The method is based on the interactive optimisation of queries by a group of test searchers using a query performance analyser. The case experiment focused on comparing the maximum effectiveness of Boolean exact-match queries, and structured and unstructured best-match queries. The experiment verified the problems in maintaining precision of Boolean queries at high recall levels. Interesting similarities were also observed between structured and unstructured best-match queries giving new light on the results of earlier studies. The case experiment showed that the proposed evaluation method yields more elaborated results in comparisons than earlier query-centred methods.

[1]  Eero Sormunen,et al.  A Method for Measuring Wide Range Performance of Boolean Queries in Full-Text Databases , 2000 .

[2]  Jaana Kekäläinen,et al.  The impact of query structure and query expansion on retrieval performance , 1998, SIGIR '98.

[3]  Jean Tague-Sutcliffe,et al.  The Pragmatics of Information Retrieval Experimentation Revisited , 1997, Inf. Process. Manag..

[4]  Jacob Shapiro,et al.  Boolean Search: Current State and Perspectives , 1999, J. Am. Soc. Inf. Sci..

[5]  S. Feldman.  Testing natural language : Comparing dialog, target and DR-link , 1996 .

[6]  Ari Pirkola,et al.  Studies on Linguistic Problems and Methods in Text Retrieval: The Effects of Anaphor and Ellipsis Resolution in Proximity Searching, and Translation and Query Structuring Methods in Cross-Language Retrieval , 1999 .

[7]  Stephen P. Harter,et al.  Online Information Retrieval: Concepts, Principles and Techniques , 1986 .

[8]  John D. Holt,et al.  Boolean System Revisited: Its Performance and its Behavior , 1995, TREC.

[9]  Ari Pirkola,et al.  The effects of query structure and dictionary setups in dictionary-based cross-language information retrieval , 1998, SIGIR '98.

[10]  Carol Tenopir,et al.  TARGET and FREESTYLE: DIALOG and Mead join the relevance ranks , 1997 .

[11]  Stephen P. Harter,et al.  Search term combinations and retrieval overlap: A proposed methodology and case study , 1990, J. Am. Soc. Inf. Sci..

[12]  Helen R. Tibbo,et al.  Freestyle vs. Boolean: A Comparison of Partial and Exact Match Retrieval Systems , 1998, Inf. Process. Manag..

[13]  Eero Sormunen,et al.  Extensions to the STAIRS Study—Empirical Evidence for the Hypothesised Ineffectiveness of Boolean Queries in Large Full-Text Databases , 2001, Information Retrieval.

[14]  M. E. Maron,et al.  An evaluation of retrieval effectiveness for a full-text document-retrieval system , 1985, CACM.

[15]  Andrew Turpin,et al.  Challenging conventional assumptions of automated information retrieval with real users: Boolean searching and batch retrieval evaluations , 2001, Inf. Process. Manag..

[16]  Eero Sormunen,et al.  Query Performance Analyser - a tool for bridging information retrieval research and instruction , 2002 .

[17]  Edward A. Fox,et al.  Research Contributions , 2014 .

[18]  William M. Shaw,et al.  Termrelevance Computations and Perfect Retrieval Performance , 1995, Inf. Process. Manag..

[19]  William R. Hersh,et al.  An evaluation of interactive Boolean and natural language searching with an online medical textbook , 1995 .

[20]  Jaana Kekäläinen,et al.  The effects of query complexity, expansion and structure on retrieval performance in probabilistic text retrieval , 1999 .

[21]  A. Rossier.  Letter to the Editor , 1986, Paraplegia.

[22]  W. Bruce Croft,et al.  Inference networks for document retrieval , 1989, SIGIR '90.

[23]  James Allan,et al.  INQUERY at TREC-5 , 1996, TREC.

[24]  William R. Hersh,et al.  An Evaluation of Interactive Boolean and Natural Language Searching with an Online Medical Textbook , 1995, J. Am. Soc. Inf. Sci..

[25]  Eero Sormunen,et al.  A novel method for the evaluation of Boolean query effectiveness across a wide operational range , 2000, SIGIR '00.

[26]  Susan Feldman.  Testing natural language , 1996 .

[27]  Douglas W. Oard,et al.  Structured translation for cross-language information retrieval , 2000, SIGIR '00.

[28]  Carol Tenopir,et al.  Magazines in Full Text: Uses and Search Strategies. , 1989 .

[29]  E. Michael Keen,et al.  Presenting Results of Experimental Retrieval Comparisons , 1997, Inf. Process. Manag..

[30]  Andrew Levison.  Ziff-Davis: sale of publishing giant impacts online industry , 1994 .

[31]  Jaana Kekäläinen,et al.  Document text characteristics affect the ranking of the most relevant documents by expanded structured queries , 2001, J. Documentation.

[32]  David A. Hull.  Using Structured Queries for Disambiguation in Cross-Language Information Retrieval , 1997 .