Modeling Information Flow in Dynamic Information Retrieval

User interaction with a dynamic information retrieval (DIR) system can be seen as a cooperative effort towards finding relevant information. In this cooperation, the user provides the system with evidence of his or her information need (e.g., in the form of queries, query reformulations, relevance judgments, or clicks). In turn, the system provides the user with evidence of the available information (e.g., in the form of a set of candidate results). Throughout this conversational process, both user and system may reduce their uncertainty with respect to each other, which may ultimately help in finding the desired information. In this paper, we present an information-theoretic model to quantify the flow of information from users to DIR systems and vice versa. By employing channels with memory and feedback, we decouple the mutual information among the behavior of the user and that of the system into directed components. As a result, we are able to measure: (i) by how much the DIR system is capable of adapting to the user; and (ii) by how much the user is influenced by the results returned by the DIR system. We discuss implications of the proposed framework for the evaluation and optimization of DIR systems.

[1]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[2]  Sekhar Tatikonda,et al.  The Capacity of Channels With Feedback , 2006, IEEE Transactions on Information Theory.

[3]  Mário S. Alvim,et al.  Measuring Information Leakage Using Generalized Gain Functions , 2012, 2012 IEEE 25th Computer Security Foundations Symposium.

[4]  Jun Wang,et al.  Dynamic Information Retrieval Modeling , 2015, Synthesis Lectures on Information Concepts, Retrieval, and Services.

[5]  Axel Legay,et al.  Hybrid Statistical Estimation of Mutual Information for Quantifying Information Flow , 2016, FM.

[6]  Mário S. Alvim,et al.  Additive and Multiplicative Notions of Leakage, and Their Capacities , 2014, 2014 IEEE 27th Computer Security Foundations Symposium.

[7]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[8]  M. de Rijke,et al.  An Introduction to Click Models for Web Search: SIGIR 2015 Tutorial , 2015, SIGIR.

[9]  Mário S. Alvim,et al.  Quantitative information flow in interactive systems , 2012, J. Comput. Secur..

[10]  Grace Hui Yang,et al.  The water filling model and the cube test: multi-dimensional evaluation for professional search , 2013, CIKM.

[11]  Rohit Chadha,et al.  Computing Information Flow Using Symbolic Model-Checking , 2014, FSTTCS.

[12]  Ryen W. White.  Interactions with Search Systems , 2016 .

[13]  ChengXiang Zhai,et al.  Estimation of statistical translation models based on mutual information for ad hoc information retrieval , 2010, SIGIR.

[14]  Jianfeng Gao,et al.  A Diversity-Promoting Objective Function for Neural Conversation Models , 2015, NAACL.

[15]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[16]  Ramji Venkataramanan,et al.  On Evaluating the Rate-Distortion Function of Sources with Feed-Forward and the Capacity of Channels with Feedback , 2007, 2007 IEEE International Symposium on Information Theory.

[17]  M. de Rijke,et al.  Click Models for Web Search , 2015, Click Models for Web Search.

[18]  Artem Grotov,et al.  Online Learning to Rank for Information Retrieval: SIGIR 2016 Tutorial , 2016, SIGIR.

[19]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[20]  Deniz Gençaga,et al.  A Recipe for the Estimation of Information Flow in a Dynamical System , 2015, Entropy.

[21]  J. Massey.  CAUSALITY, FEEDBACK AND DIRECTED INFORMATION , 1990 .