Evaluating sources of implicit feedback in web searches

The study investigates the relationship between the types of behavior that can be captured from Web searches and searchers' interests. Web search cases which involve underspecification of information needs at the beginning and modification of search strategies during the search process will be collected and examined by human analysts. The study focuses on identifying the rules used by analysts to infer searcher interests. These rules can be put into algorithms as the basis for systems that provide query modification suggestions or implicitly reformulate the query as the searcher continues to work.

[1]  Christoph Hölscher,et al.  Web search behavior of Internet experts and newbies , 2000, Comput. Networks.

[2]  Beth Sandore,et al.  Applying the results of transaction log analysis , 1993 .

[3]  Norbert Fuhr,et al.  Active Support for Query Formulation in Virtual Digital Libraries: A Case Study with DAFFODIL , 2005, ECDL.

[4]  Ryen W. White,et al.  Finding relevant documents using top ranking sentences: an evaluation of two alternative schemes , 2002, SIGIR '02.

[5]  Nicholas J. Belkin,et al.  Display time as implicit feedback: understanding task effects , 2004, SIGIR '04.

[6]  Ernesto Arroyo,et al.  Usability tool for analysis of web designs using mouse tracks , 2006, CHI Extended Abstracts.

[7]  M. Posner,et al.  Orienting of Attention* , 1980, The Quarterly journal of experimental psychology.

[8]  Nicholas J. Belkin,et al.  Interaction with Texts: Information Retrieval as Information-Seeking Behavior , 1993, Information Retrieval.

[9]  Karen Michelle O'Keefe The quality of medical students' confidence judgments when using external information resources : the effects of different media formats, source of questions, and question formats , 2000 .

[10]  Hideo Joho,et al.  Hierarchical presentation of expansion terms , 2002, SAC '02.

[11]  Martin Kurth,et al.  The limits and limitations of transaction log analysis , 1993 .

[12]  Albrecht Schmidt,et al.  Knowing the User's Every Move – User Activity Tracking for Website Usability Evaluation and Implicit Interaction , 2006 .

[13]  Anna L. Cox,et al.  The Role of Mouse Movements in Interactive Search , 2006 .

[14]  Ryen W. White,et al.  A study of factors affecting the utility of implicit relevance feedback , 2005, SIGIR '05.

[15]  W. Bruce Croft,et al.  Predicting query performance , 2002, SIGIR '02.

[16]  Paul P. Maglio,et al.  Attentive agents , 2003, Commun. ACM.

[17]  ChengXiang Zhai,et al.  Active feedback in ad hoc information retrieval , 2005, SIGIR '05.

[18]  Peter Ingwersen,et al.  Search Procedures in the Library - Analysed from the Cognitive Point of View , 1982, J. Documentation.

[19]  K. Rayner Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[20]  Gary Marchionini,et al.  Previews and overviews in digital libraries: designing surrogates to support visual information seeking , 2000 .

[21]  Susan T. Dumais,et al.  Personalizing Search via Automated Analysis of Interests and Activities , 2005, SIGIR.

[22]  Douglas W. Oard,et al.  Modeling Information Content Using Observable Behavior , 2001 .

[23]  Ian Ruthven,et al.  Re-examining the potential effectiveness of interactive query expansion , 2003, SIGIR.

[24]  Oren Etzioni,et al.  Grouper: A Dynamic Clustering Interface to Web Search Results , 1999, Comput. Networks.

[25]  Susan T. Dumais,et al.  The vocabulary problem in human-system communication , 1987, CACM.

[26]  Wendy T. Lucas,et al.  Form and function: The impact of query term and operator usage on Web search results , 2002, J. Assoc. Inf. Sci. Technol..

[27]  Susan T. Dumais,et al.  Optimizing search by showing results in context , 2001, CHI.

[28]  Chris Buckley,et al.  Improving automatic query expansion , 1998, SIGIR '98.

[29]  Iwan G. J. H. Wopereis,et al.  Differences between novice and experienced users in searching information on the World Wide Web , 2000, J. Am. Soc. Inf. Sci..

[30]  Wei Zhang,et al.  An Iterative Implicit Feedback Approach to Personalized Search , 2006, ACL.

[31]  John M. Carroll,et al.  The role of laboratory experiments in HCI: help, hindrance, or ho-hum? , 1989, CHI '89.

[32]  Qi Li,et al.  UMass at TREC 2003: HARD and QA , 2003, TREC.

[33]  Barry Smyth,et al.  Passive Profiling from Server Logs in an Online Recruitment Environment , 2001, IJCAI 2001.

[34]  Jaime Teevan,et al.  Implicit feedback for inferring user preference: a bibliography , 2003, SIGF.

[35]  Barry Smyth,et al.  Exploiting Query Repetition and Regularity in an Adaptive Community-Based Web Search Engine , 2004, User Modeling and User-Adapted Interaction.

[36]  Yi Zhang,et al.  Combining Multiple Forms of Evidence While Filtering , 2005, HLT.

[37]  Tom Peters,et al.  The history and development of transaction log analysis , 1993 .

[38]  John R. Anderson,et al.  What can a mouse cursor tell us more?: correlation of eye/mouse movements on web browsing , 2001, CHI Extended Abstracts.

[39]  Joseph W. Janes,et al.  Relevance judgments and the incremental presentation of document representations , 1991, Inf. Process. Manag..

[40]  ChengXiang Zhai,et al.  Implicit user modeling for personalized search , 2005, CIKM '05.

[41]  Eszter Hargittai,et al.  Beyond logs and surveys: In-depth measures of people's web use skills , 2002, J. Assoc. Inf. Sci. Technol..

[42]  Susan T. Dumais,et al.  Improving Web Search Ranking by Incorporating User Behavior Information , 2019, SIGIR Forum.

[43]  Yoichi Shinoda,et al.  Information filtering based on user behavior analysis and best match text retrieval , 1994, SIGIR '94.

[44]  Pertti Vakkari,et al.  Cognition and changes of search terms and tactics during task performance: A longitudinal case study , 2000, RIAO.

[45]  Andrew T Duchowski,et al.  A breadth-first survey of eye-tracking applications , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[46]  Peter Ingwersen,et al.  Information Retrieval Interaction , 1992 .

[47]  Eugene Volokh,et al.  Personalization and privacy , 2000, CACM.

[48]  Steve Fox,et al.  Evaluating implicit measures to improve web search , 2005, TOIS.

[49]  Joseph H. Goldberg,et al.  Eye tracking in web search tasks: design implications , 2002, ETRA.

[50]  Andrew Monk,et al.  Improving Your Human-Computer Interface: A Practical Technique , 1993 .

[51]  Filip Radlinski,et al.  Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search , 2007, TOIS.

[52]  ChengXiang Zhai,et al.  Active Feedback - UIUC TREC-2003 HARD Experiments , 2003, TREC.

[53]  Saul Greenberg,et al.  Revisitation patterns in World Wide Web navigation , 1997, CHI.

[54]  James E. Pitkow,et al.  Characterizing Browsing Strategies in the World-Wide Web , 1995, Comput. Networks ISDN Syst..

[55]  Thorsten Joachims,et al.  Accurately Interpreting Clickthrough Data as Implicit Feedback , 2017 .

[56]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[57]  Bryce Allen,et al.  Content Analysis in Library and Information Science Research. , 1990 .

[58]  Nicholas J. Belkin,et al.  Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval , 2001, Inf. Process. Manag..

[59]  Douglas W. Oard,et al.  User Modeling for Information Access Based on Implicit Feedback , 2000 .

[60]  Elaine Toms,et al.  An approach to search for the digital library , 2001, JCDL '01.

[61]  Jillian R. Griffiths,et al.  An improved method of studying user-system interaction by combining transaction log analysis and protocol analysis , 2002, Inf. Res..

[62]  Ivan Koychev,et al.  Within-Document Retrieval: A User-Centred Evaluation of Relevance Profiling , 2004, Information Retrieval.

[63]  Rosie Jones,et al.  Query word deletion prediction , 2003, SIGIR.

[64]  Kerry Rodden,et al.  Exploring How Mouse Movements Relate to Eye Movements on Web Search Results Pages , 2007 .

[65]  Samuel Kaski,et al.  Can Relevance be Inferred from Eye Movements in Information Retrieval , 2003 .

[66]  Marcia J. Bates,et al.  The design of browsing and berrypicking techniques for the online search interface , 1989 .

[67]  Joseph H. Goldberg,et al.  Computer interface evaluation using eye movements: methods and constructs , 1999 .

[68]  Ophir Frieder,et al.  Hourly analysis of a very large topically categorized web query log , 2004, SIGIR '04.

[69]  Susan T. Dumais,et al.  Evaluating implicit measures to improve the search experiences , 2003 .

[70]  Pertti Vakkari,et al.  Task-based information searching , 2005, Annu. Rev. Inf. Sci. Technol..

[71]  Susan T. Dumais,et al.  Learning user interaction models for predicting web search result preferences , 2006, SIGIR.

[72]  Nicholas J. Belkin,et al.  Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems , 1995 .

[73]  Nicholas J. Belkin,et al.  Distributed expert problem treatment as a model for information system analysis and design , 1982 .

[74]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[75]  Ryen W. White,et al.  A study on the effects of personalization and task information on implicit feedback performance , 2006, CIKM '06.

[76]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[77]  Andrea Lockerd Thomaz,et al.  Cheese: tracking mouse movement activity on websites, a tool for user modeling , 2001, CHI Extended Abstracts.

[78]  Barbara M. Wildemuth,et al.  Effective methods for studying information seeking and use , 2002, J. Assoc. Inf. Sci. Technol..

[79]  Diane Kelly,et al.  IMPLICIT FEEDBACK: USING BEHAVIOR TO INFER RELEVANCE , 2005 .

[80]  Carol Collier Kuhlthau,et al.  Seeking Meaning: a process approach to library and information services" Ablex Publishing , 2003 .

[81]  Bradley N. Miller,et al.  GroupLens: applying collaborative filtering to Usenet news , 1997, CACM.

[82]  Stephen G. Eick,et al.  Seesoft-A Tool For Visualizing Line Oriented Software Statistics , 1992, IEEE Trans. Software Eng..

[83]  Rico Fischer,et al.  Eye tracking and its application in usability and media research , 2003, MMI Interakt..

[84]  Peter Ingwersen,et al.  Cognitive Perspectives of Information Retrieval Interaction: Elements of a Cognitive IR Theory , 1996, J. Documentation.

[85]  Peter G. Anick Using terminological feedback for web search refinement: a log-based study , 2003, SIGIR.

[86]  Pia Borlund,et al.  Experimental components for the evaluation of interactive information retrieval systems , 2000, J. Documentation.

[87]  Xia Lin Map displays for information retrieval , 1997 .

[88]  Jonathan L. Herlocker,et al.  Click data as implicit relevance feedback in web search , 2007, Inf. Process. Manag..

[89]  Bernard J. Jansen,et al.  Search log analysis: What it is, what's been done, how to do it , 2006 .

[90]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[91]  Mounia Lalmas,et al.  A survey on the use of relevance feedback for information access systems , 2003, The Knowledge Engineering Review.

[92]  H. Deubel,et al.  Saccade target selection and object recognition: Evidence for a common attentional mechanism , 1996, Vision Research.

[93]  Ryen W. White,et al.  The Use of Implicit Evidence for Relevance Feedback in Web Retrieval , 2002, ECIR.

[94]  Mark Claypool,et al.  Implicit interest indicators , 2001, IUI '01.

[95]  Chirag Shah,et al.  Using collaborative queries to improve retrieval for difficult topics , 2007, SIGIR.

[96]  Pertti Vakkari,et al.  Students' conceptual structure, search process, and outcome while preparing a research proposal: A longitudinal case study , 2003, J. Assoc. Inf. Sci. Technol..

[97]  Xuehua Shen,et al.  Context-sensitive information retrieval using implicit feedback , 2005, SIGIR '05.

[98]  Amanda Spink,et al.  Real life, real users, and real needs: a study and analysis of user queries on the web , 2000, Inf. Process. Manag..

[99]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[100]  John R. Anderson,et al.  Eye tracking the visual search of click-down menus , 1999, CHI '99.

[101]  Paul P. Maglio,et al.  SUITOR: an attentive information system , 2000, IUI '00.

[102]  Nicholas J. Belkin,et al.  Reading time, scrolling and interaction: exploring implicit sources of user preferences for relevance feedback , 2001, Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.

[103]  Young-Woo Seo,et al.  Personalized web-document filtering using reinforcement learning , 2001, Appl. Artif. Intell..

[104]  Brian Detlor,et al.  Information Seeking on the Web: An Integrated Model of Browsing and Searching , 2000, First Monday.

[105]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction (4th Edition) , 2004 .

[106]  Dania Bilal Children's use of the Yahooligans! Web search engine: I. Cognitive, physical, and affective behaviors on fact‐based search tasks , 2000 .

[107]  Barry Smyth,et al.  Collecting community wisdom: integrating social search & social navigation , 2007, IUI '07.

[108]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[109]  Ingrid Hsieh-Yee Research on Web Search Behavior. , 2001 .

[110]  Gary Marchionini,et al.  Information Seeking Behavior on Statistical Websites: Theoretical and Design Implications , 1998 .

[111]  Yoshinori Hijikata,et al.  Implicit user profiling for on demand relevance feedback , 2004, IUI '04.

[112]  Dongheng Li,et al.  openEyes: a low-cost head-mounted eye-tracking solution , 2006, ETRA.

[113]  Robert J. K. Jacob,et al.  Evaluation of eye gaze interaction , 2000, CHI.

[114]  Benjamin Rey,et al.  Generating query substitutions , 2006, WWW '06.

[115]  Diane Kelly,et al.  Questionnaire mode effects in interactive information retrieval experiments , 2008, Inf. Process. Manag..

[116]  Carol Tenopir,et al.  Users' interaction with World Wide Web resources: an exploratory study using a holistic approach , 2000, Inf. Process. Manag..

[117]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[118]  Jude W. Shavlik,et al.  Learning users' interests by unobtrusively observing their normal behavior , 2000, IUI '00.

[119]  Gary Marchionini,et al.  Information Seeking in Electronic Environments , 1995 .

[120]  Bill N. Schilit,et al.  From reading to retrieval: freeform ink annotations as queries , 1999, SIGIR '99.

[121]  Micheal D. Cooper Predicting the relevance of a library catalog search , 2001 .

[122]  Diane Kelly Understanding implicit feedback and document preference: a naturalistic user study , 2004, SIGF.

[123]  Peiling Wang,et al.  Mining longitudinal web queries: Trends and patterns , 2003, J. Assoc. Inf. Sci. Technol..

[124]  Amos Tversky,et al.  On the evaluation of probability judgments : calibration, resolution, and monotonicity , 1993 .

[125]  Xin Fu,et al.  Elicitation of term relevance feedback: an investigation of term source and context , 2006, SIGIR.

[126]  Catherine C. Marshall,et al.  Turning the page on navigation , 2005, Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05).

[127]  Ryen W. White Implicit feedback for interactive information retrieval , 2005, SIGF.

[128]  Amanda Spink,et al.  Web Search: Public Searching of the Web , 2011, Information Science and Knowledge Management.

[129]  Paul B. Kantor,et al.  Capturing human intelligence in the net , 2000, CACM.

[130]  Gary Marchionini Information-seeking strategies of novices using a full-text electronic encyclopedia , 1989 .

[131]  Frank Curtis Stevens,et al.  Knowledge-based assistance for accessing large, poorly structured information spaces , 1993 .

[132]  Andrew T. Duchowski,et al.  Using Eye Tracking to Evaluate Alternative Search Results Interfaces , 2005 .

[133]  Samuel Kaski,et al.  Combining eye movements and collaborative filtering for proactive information retrieval , 2005, SIGIR '05.

[134]  Gerard Salton,et al.  Improving Retrieval Performance by Relevance Feedback , 1997 .

[135]  Gary Marchionini,et al.  Examining the effectiveness of real-time query expansion , 2007, Inf. Process. Manag..

[136]  Ben Shneiderman,et al.  Facilitating data exploration with query previews: A study of user performance and preference , 2000, Behav. Inf. Technol..

[137]  Thorsten Joachims,et al.  Eye-tracking analysis of user behavior in WWW search , 2004, SIGIR '04.

[138]  Ellen M. Voorhees,et al.  Overview of the TREC 2004 Robust Retrieval Track , 2004 .

[139]  Eric Brill,et al.  Improving web search ranking by incorporating user behavior information , 2006, SIGIR.

[140]  Bernard J. Jansen,et al.  A review of web searching studies and a framework for future research , 2001 .

[141]  Gary Marchionini,et al.  User studies informing E-table interfaces , 2003, Inf. Process. Manag..