User Simulations for Interactive Search: Evaluating Personalized Query Suggestion

In this paper, we address the question “what is the influence of user search behaviour on the effectiveness of personalized query suggestion?”. We implemented a method for query suggestion that generates candidate follow-up queries from the documents clicked by the user. This is a potentially effective method for query suggestion, but it heavily depends on user behaviour. We set up a series of experiments in which we simulate a large range of user session behaviour to investigate its influence. We found that query suggestion is not profitable for all user types. We identified a number of significant effects of user behaviour on session effectiveness. In general, it appears that there is extensive interplay between the examination behaviour, the term selection behaviour, the clicking behaviour and the query modification strategy. The results suggest that query suggestion strategies need to be adapted to specific user behaviours.

[1]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR '00.

[2]  Kalervo Järvelin.  Interactive relevance feedback with graded relevance and sentence extraction: simulated user experiments , 2009, CIKM.

[3]  W. Bruce Croft,et al.  Automatic boolean query suggestion for professional search , 2011, SIGIR.

[4]  Mark D. Smucker,et al.  Report on the SIGIR 2010 workshop on the simulation of interaction , 2011, SIGF.

[5]  Peter Ingwersen,et al.  Developing a Test Collection for the Evaluation of Integrated Search , 2010, ECIR.

[6]  James Allan,et al.  Task-aware query recommendation , 2013, SIGIR.

[7]  Leif Azzopardi,et al.  How query cost affects search behavior , 2013, SIGIR.

[8]  Edward Cutrell,et al.  An eye tracking study of the effect of target rank on web search , 2007, CHI.

[9]  Kalervo Järvelin,et al.  Test Collection-Based IR Evaluation Needs Extension toward Sessions - A Case of Extremely Short Queries , 2009, AIRS.

[10]  Kalervo Järvelin,et al.  Time drives interaction: simulating sessions in diverse searching environments , 2012, SIGIR '12.

[11]  Nicholas J. Belkin,et al.  Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval , 2001, Inf. Process. Manag..

[12]  Tie-Yan Liu,et al.  Information Retrieval Technology , 2014, Lecture Notes in Computer Science.

[13]  Thorsten Joachims,et al.  Accurately interpreting clickthrough data as implicit feedback , 2005, SIGIR '05.

[14]  Marcia J. Bates,et al.  Information search tactics , 1979, J. Am. Soc. Inf. Sci..

[15]  Matthew Hurst,et al.  A Language Model Approach to Keyphrase Extraction , 2003, ACL 2003.

[16]  Yen-Jen Oyang,et al.  Relevant term suggestion in interactive web search based on contextual information in query session logs , 2003, J. Assoc. Inf. Sci. Technol..

[17]  ChengXiang Zhai,et al.  Implicit user modeling for personalized search , 2005, CIKM '05.

[18]  Prasenjit Mitra,et al.  Query suggestions in the absence of query logs , 2011, SIGIR.

[19]  Chao Liu,et al.  Click chain model in web search , 2009, WWW '09.

[20]  Katja Hofmann,et al.  Reusing historical interaction data for faster online learning to rank for IR , 2013, DIR.

[21]  Wessel Kraaij,et al.  Query Term Suggestion in Academic Search , 2014, ECIR.