Generating Simulated Relevance Feedback: A Prognostic Search approach

Implicit relevance feedback has proved to be a important resource in improving search accuracy and personalization. However, researchers who rely on feedback data for testing their algorithms or other personalization related problems are loomed with problems like unavailability of data, staling up of data and so on. Given these problems, we are motivated towards creating a synthetic user relevance feedback data, based on insights from query log analysis. We call this simulated feedback. We believe that simulated feedback can be immensely beneficial to web search engine and personalization research communities by greatly reducing efforts involved in collecting user feedback. The benefits from "Simulated feedback" are - it is easy to obtain and also the process of obtaining the feedback data is repeatable, customizable and does not need the interactions of the user. In this paper, we describe a simple yet effective approach for creating simulated feedback. We have evaluated our system using the clickthrough data of the users and achieved 77% accuracy in generating click-through data.

[1]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[2]  ChengXiang Zhai,et al.  Implicit user modeling for personalized search , 2005, CIKM '05.

[3]  Feng Qiu,et al.  Automatic identification of user interest for personalized search , 2006, WWW '06.

[4]  Javed Mostafa,et al.  Simulation Studies of Different Dimensions of Users' Interests and their Impact on User Modeling and Information Filtering , 2003, Information Retrieval.

[5]  Daniel E. Rose,et al.  Understanding user goals in web search , 2004, WWW '04.

[6]  Nicholas J. Belkin,et al.  Reading time, scrolling and interaction: exploring implicit sources of user preferences for relevance feedback , 2001, Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.

[7]  Nick Craswell,et al.  An experimental comparison of click position-bias models , 2008, WSDM '08.

[8]  Ryen W. White,et al.  Evaluating implicit feedback models using searcher simulations , 2005, TOIS.

[9]  Danielle S. McNamara,et al.  Handbook of latent semantic analysis , 2007 .

[10]  Masatoshi Yoshikawa,et al.  Adaptive web search based on user profile constructed without any effort from users , 2004, WWW '04.

[11]  Gerard Salton,et al.  Improving retrieval performance by relevance feedback , 1997, J. Am. Soc. Inf. Sci..

[12]  Thorsten Joachims,et al.  Eye-tracking analysis of user behavior in WWW search , 2004, SIGIR '04.

[13]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[14]  Chao Liu,et al.  Efficient multiple-click models in web search , 2009, WSDM '09.

[15]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[16]  Olivier Chapelle,et al.  A dynamic bayesian network click model for web search ranking , 2009, WWW '09.

[17]  Thorsten Joachims,et al.  Accurately interpreting clickthrough data as implicit feedback , 2005, SIGIR '05.

[18]  Donna K. Harman,et al.  Relevance feedback revisited , 1992, SIGIR '92.

[19]  Steve Fox,et al.  Evaluating implicit measures to improve web search , 2005, TOIS.

[20]  Mark Claypool,et al.  Implicit interest indicators , 2001, IUI '01.

[21]  Susan T. Dumais,et al.  Learning user interaction models for predicting web search result preferences , 2006, SIGIR.

[22]  Filip Radlinski,et al.  Evaluating the Robustness of Learning from Implicit Feedback , 2006, ArXiv.