Subjective and objective evaluation of interactive and automatic query expansion

Purpose – Query expansion and query limitation are two known techniques for assisting users to define efficient queries. The purpose of this article is to examine the effectiveness of the two methods.Design/methodology/approach – The research entailed an objective and subjective evaluation of the effectiveness of automatic and interactive query expansion and of two query limit options. The evaluation included both lab simulations and large‐scale user studies. The objective aspects were evaluated in lab simulations with experts judging user performance. The subjective analysis was carried out by having the participants evaluate the quality of, and express their satisfaction with, the retrieval process and its results, thus employing perceived‐value analysis.Findings – The main findings reveal a difference between the perceived and real values of these techniques. While users expressed their satisfaction with interactive query expansion and its performance, the real‐value analysis of their performance did n...

[1]  Justin Zobel,et al.  Questioning Query Expansion: An Examination of Behaviour and Parameters , 2004, ADC.

[2]  Stephen E. Robertson,et al.  Flexible pseudo-relevance feedback using optimization tables , 2001, SIGIR '01.

[3]  James Allan,et al.  Relevance feedback with too much data , 1995, SIGIR '95.

[4]  Peter Willett,et al.  The limitations of term co-occurrence data for query expansion in document retrieval systems , 1991, J. Am. Soc. Inf. Sci..

[5]  Susan T. Dumais,et al.  Learned Vector-Space Models for Document Retrieval , 1995, Inf. Process. Manag..

[6]  Stephen E. Robertson,et al.  Microsoft Cambridge at TREC-9: Filtering Track , 2000, TREC.

[7]  Nicholas J. Belkin,et al.  Relevance Feedback versus Local Context Analysis as Term Suggestion Devices: Rutgers' TREC-8 Interactive Track Experience , 1999, TREC.

[8]  Wendy T. Lucas,et al.  Form and function: The impact of query term and operator usage on Web search results , 2002, J. Assoc. Inf. Sci. Technol..

[9]  Monika Henzinger,et al.  Analysis of a very large web search engine query log , 1999, SIGF.

[10]  J. D. Bovey,et al.  Weighting, ranking and relevance feedback in a front—end system , 1986, J. Inf. Sci..

[11]  Ingrid Hsieh-Yee,et al.  Effects of Search Experience and Subject Knowledge on the Search Tactics of Novice and Experienced Searchers. , 1993 .

[12]  Amanda Spink,et al.  From E-Sex to E-Commerce: Web Search Changes , 2002, Computer.

[13]  Christoph Hölscher,et al.  Web search behavior of Internet experts and newbies , 2000, Comput. Networks.

[14]  Claudio Carpineto,et al.  An information-theoretic approach to automatic query expansion , 2001, TOIS.

[15]  Mark Magennis,et al.  The potential and actual effectiveness of interactive query expansion , 1997, SIGIR '97.

[16]  Alan F. Smeaton,et al.  The Retrieval Effects of Query Expansion on a Feedback Document Retrieval System , 1983, Comput. J..

[17]  W. Bruce Croft,et al.  Query expansion using local and global document analysis , 1996, SIGIR '96.

[18]  Aviezri S. Fraenkel,et al.  Local Feedback in Full-Text Retrieval Systems , 1977, JACM.

[19]  Nicholas J. Belkin,et al.  A case for interaction: a study of interactive information retrieval behavior and effectiveness , 1996, CHI.

[20]  James Allan,et al.  Automatic Query Expansion Using SMART: TREC 3 , 1994, TREC.

[21]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[22]  Karen Spärck Jones,et al.  The use of automatically-obtained keyword classifications for information retrieval , 1969, Inf. Storage Retr..

[23]  Riao.  RIAO 94 Conference proceedings : Intelligent Multimedia Information Retrieval Systems and Management, Rockefeller University, New York, October 11-13, 1994 , 1994 .

[24]  Donna K. Harman,et al.  Relevance feedback revisited , 1992, SIGIR '92.

[25]  Keiichiro Hoashi,et al.  Query expansion based on predictive algorithms for collaborative filtering , 2001, Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.

[26]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[27]  Stephen E. Robertson,et al.  Microsoft Cambridge at TREC 2002: Filtering Track , 2002, TREC.

[28]  Andrew Turpin,et al.  Do batch and user evaluations give the same results? , 2000, SIGIR '00.

[29]  Bracha Shapira,et al.  Evaluation of the real and perceived value of automatic and interactive query expansion , 2004, SIGIR '04.

[30]  Micheline Beaulieu,et al.  Experiments on interfaces to support query expansion , 1997, J. Documentation.

[31]  Gerard Salton,et al.  Improving retrieval performance by relevance feedback , 1997, J. Am. Soc. Inf. Sci..

[32]  Justin Zobel,et al.  When query expansion fails , 2003, SIGIR '03.

[33]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[34]  Ian Ruthven,et al.  Re-examining the potential effectiveness of interactive query expansion , 2003, SIGIR.

[35]  Mark Sanderson,et al.  A Study of User Interaction with a Concept-Based Interactive Query Expansion Support Tool , 2004, ECIR.

[36]  Chris Buckley,et al.  Improving automatic query expansion , 1998, SIGIR '98.

[37]  Amanda Spink,et al.  Real life information retrieval: a study of user queries on the Web , 1998, SIGF.

[38]  Amanda Spink,et al.  Real life, real users, and real needs: a study and analysis of user queries on the web , 2000, Inf. Process. Manag..

[39]  W. Bruce Croft,et al.  An Association Thesaurus for Information Retrieval , 1994, RIAO.