TellMyRelevance!: predicting the relevance of web search results from cursor interactions

It is crucial for the success of a search-driven web application to answer users' queries in the best possible way. A common approach is to use click models for guessing the relevance of search results. However, these models are imprecise and waive valuable information one can gain from non-click user interactions. We introduce TellMyRelevance!---a novel automatic end-to-end pipeline for tracking cursor interactions at the client, analyzing these and learning according relevance models. Yet, the models depend on the layout of the search results page involved, which makes them difficult to evaluate and compare. Thus, we use a Random Mouse Cursor as an extension to our pipeline for generating layout-dependent baselines. Based on these, we can perform evaluations of real-world relevance models. A large-scale interaction log analysis showed that we can learn relevance models whose predictions compare favorably to predictions of an existing state-of-the-art click model.

[1]  Ryen W. White,et al.  No clicks, no problem: using cursor movements to understand and improve search , 2011, CHI.

[2]  Eugene Agichtein,et al.  ViewSer: enabling large-scale remote user studies of web search examination and interaction , 2011, SIGIR.

[3]  John R. Anderson,et al.  What can a mouse cursor tell us more?: correlation of eye/mouse movements on web browsing , 2001, CHI Extended Abstracts.

[4]  Pierre Baldi,et al.  Assessing the accuracy of prediction algorithms for classification: an overview , 2000, Bioinform..

[5]  Alessandro Bozzon,et al.  A Characterization of the Layout Definition Problem for Web Search Results , 2010, OTM Workshops.

[6]  Lydia B. Chilton,et al.  Addressing people's information needs directly in a web search result page , 2011, WWW.

[7]  Ryen W. White,et al.  Improving searcher models using mouse cursor activity , 2012, SIGIR '12.

[9]  Eugene Agichtein,et al.  Detecting success in mobile search from interaction , 2011, SIGIR '11.

[10]  Ernesto Arroyo,et al.  Usability tool for analysis of web designs using mouse tracks , 2006, CHI Extended Abstracts.

[11]  Eugene Agichtein,et al.  Beyond dwell time: estimating document relevance from cursor movements and other post-click searcher behavior , 2012, WWW.

[12]  Chao Liu,et al.  BBM: bayesian browsing model from petabyte-scale data , 2009, KDD.

[13]  Yisong Yue,et al.  Beyond position bias: examining result attractiveness as a source of presentation bias in clickthrough data , 2010, WWW '10.

[14]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[15]  Eibe Frank,et al.  A Simple Approach to Ordinal Classification , 2001, ECML.

[16]  Benjamin Piwowarski,et al.  A user browsing model to predict search engine click data from past observations. , 2008, SIGIR '08.

[17]  Olivier Chapelle,et al.  A dynamic bayesian network click model for web search ranking , 2009, WWW '09.

[18]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[19]  Nick Craswell,et al.  An experimental comparison of click position-bias models , 2008, WSDM '08.

[20]  Ramakrishnan Srikant,et al.  User browsing models: relevance versus examination , 2010, KDD.

[21]  Qiang Yang,et al.  A Whole Page Click Model to Better Interpret Search Engine Click Data , 2011, AAAI.

[22]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[23]  Elizabeth F. Churchill,et al.  Mouse tracking: measuring and predicting users' experience of web-based content , 2012, CHI.

[24]  Chao Liu,et al.  Efficient multiple-click models in web search , 2009, WSDM '09.