Modeling behavioral factors ininteractive information retrieval

In real-life, information retrieval consists of sessions of one or more query iterations. Each iteration has several subtasks like query formulation, result scanning, document link clicking, document reading and judgment, and stopping. Each of the subtasks has behavioral factors associated with them. These factors include search goals and cost constraints, query formulation strategies, scanning and stopping strategies, and relevance assessment behav-ior. Traditional IR evaluation focuses on retrieval and result presentation methods, and interaction within a single-query session. In the present study we aim at assessing the effects of the behavioral factors on retrieval effectiveness. Our research questions include how effective is human behavior employing search strategies compared to various baselines under various search goals and time constraints. We examine both ideal as well as fallible human behavior and wish to identify robust behaviors, if any. Methodologically, we use extensive simulation of human behavior in a test collection. Our findings include that (a) human behavior using multi-query sessions may exceed in effectiveness comparable single-query sessions, (b) the same empirically observed behavioral patterns are reasonably effective under various search goals and constraints, but (c) remain on average clearly below the best possible ones. Moreover, there is no behavioral pattern for sessions that would be even close to winning in most cases; the information need (or topic) in relation to the test collection is a determining factor.

[1]  Benjamin Piwowarski,et al.  Model Based Comparison of Discounted Cumulative Gain and Average Precision , 2013, J. Discrete Algorithms.

[2]  Pertti Vakkari,et al.  The influence of relevance levels on the effectiveness of interactive information retrieval , 2004, J. Assoc. Inf. Sci. Technol..

[3]  J. Shane Culpepper,et al.  Including summaries in system evaluation , 2009, SIGIR.

[4]  Marcia J. Bates,et al.  Information search tactics , 1979, J. Am. Soc. Inf. Sci..

[5]  Donna K. Harman,et al.  Relevance feedback revisited , 1992, SIGIR '92.

[6]  Kalervo Järvelin,et al.  Simulating Simple and Fallible Relevance Feedback , 2011, ECIR.

[7]  Catherine L. Smith,et al.  User adaptation: good results from poor systems , 2008, SIGIR '08.

[8]  Leif Azzopardi,et al.  The economics in interactive information retrieval , 2011, SIGIR.

[9]  Diane Kelly,et al.  Methods for Evaluating Interactive Information Retrieval Systems with Users , 2009, Found. Trends Inf. Retr..

[10]  Peter Ingwersen,et al.  The Turn - Integration of Information Seeking and Retrieval in Context , 2005, The Kluwer International Series on Information Retrieval.

[11]  Eero Sormunen,et al.  Liberal relevance criteria of TREC -: counting on negligible documents? , 2002, SIGIR '02.

[12]  Tetsuya Sakai.  Give me just one highly relevant document: P-measure , 2006, SIGIR '06.

[13]  Ben Carterette,et al.  Simulating simple user behavior for system effectiveness evaluation , 2011, CIKM '11.

[14]  Kalervo Järvelin,et al.  Test Collection-Based IR Evaluation Needs Extension toward Sessions - A Case of Extremely Short Queries , 2009, AIRS.

[15]  Mark D. Smucker,et al.  Report on the SIGIR 2010 workshop on the simulation of interaction , 2011, SIGF.

[16]  Mark D. Smucker.  Towards Timed Predictions of Human Performance for Interactive Information Retrieval Evaluation , 2009 .

[17]  Charles L. A. Clarke,et al.  Time-based calibration of effectiveness measures , 2012, SIGIR '12.

[18]  Ben Carterette,et al.  Incorporating variability in user behavior into systems based evaluation , 2012, CIKM.

[19]  Kalervo Järvelin,et al.  Time drives interaction: simulating sessions in diverse searching environments , 2012, SIGIR '12.