Segment-level display time as implicit feedback: a comparison to eye tracking

We examine two basic sources for implicit relevance feedback on the segment level for search personalization: eye tracking and display time. A controlled study has been conducted where 32 participants had to view documents in front of an eye tracker, query a search engine, and give explicit relevance ratings for the results. We examined the performance of the basic implicit feedback methods with respect to improved ranking and compared their performance to a pseudo relevance feedback baseline on the segment level and the original ranking of a Web search engine. Our results show that feedback based on display time on the segment level is much coarser than feedback from eye tracking. But surprisingly, for re-ranking and query expansion it did work as well as eye-tracking-based feedback. All behavior-based methods performed significantly better than our non-behavior-based baseline and especially improved poor initial rankings of the Web search engine. The study shows that segment-level display time yields comparable results as eye-tracking-based feedback. Thus, it should be considered in future personalization systems as an inexpensive but precise method for implicit feedback.

[1]  Andreas Dengel,et al.  Eye movements as implicit relevance feedback , 2008, CHI Extended Abstracts.

[2]  Gerard Salton,et al.  The SMART Retrieval System , 1971 .

[3]  Karl Gyllstrom,et al.  Seeing is retrieving: building information context from what the user sees , 2008, IUI '08.

[4]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[5]  Susan T. Dumais,et al.  Improving Web Search Ranking by Incorporating User Behavior Information , 2019, SIGIR Forum.

[6]  Yoichi Shinoda,et al.  Information filtering based on user behavior analysis and best match text retrieval , 1994, SIGIR '94.

[7]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR '00.

[8]  Bill N. Schilit,et al.  From reading to retrieval: freeform ink annotations as queries , 1999, SIGIR '99.

[9]  Nicholas J. Belkin,et al.  Some(what) grand challenges for information retrieval , 2008, SIGF.

[10]  Qi Li,et al.  Personalized web exploration with task models , 2008, WWW.

[11]  Douglas W. Oard,et al.  Modeling Information Content Using Observable Behavior , 2001 .

[12]  Joemon M. Jose,et al.  Affective feedback: an investigation into the role of emotions in the information seeking process , 2008, SIGIR '08.

[13]  Steve Fox,et al.  Evaluating implicit measures to improve web search , 2005, TOIS.

[14]  Mark Claypool,et al.  Implicit interest indicators , 2001, IUI '01.

[15]  Ryen W. White,et al.  Discovering Hidden Contextual Factors for Implicit Feedback , 2007, CIR.

[16]  Nicholas J. Belkin,et al.  Display time as implicit feedback: understanding task effects , 2004, SIGIR '04.

[17]  Samuel Kaski,et al.  Implicit Relevance Feedback from Eye Movements , 2005, ICANN.

[18]  Andreas Dengel,et al.  Query expansion using gaze-based feedback on the subdocument level , 2008, SIGIR '08.

[19]  Barry Smyth,et al.  Passive Profiling from Server Logs in an Online Recruitment Environment , 2001, IJCAI 2001.

[20]  Jaime Teevan,et al.  Implicit feedback for inferring user preference: a bibliography , 2003, SIGF.

[21]  Nicholas J. Belkin,et al.  Reading time, scrolling and interaction: exploring implicit sources of user preferences for relevance feedback , 2001, Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.

[22]  Filip Radlinski,et al.  Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search , 2007, TOIS.

[23]  Ryen W. White,et al.  A study on the effects of personalization and task information on implicit feedback performance , 2006, CIKM '06.