To personalize or not to personalize: modeling queries with variation in user intent

In most previous work on personalized search algorithms, the results for all queries are personalized in the same manner. However, as we show in this paper, there is a lot of variation across queries in the benefits that can be achieved through personalization. For some queries, everyone who issues the query is looking for the same thing. For other queries, different people want very different results even though they express their need in the same way. We examine variability in user intent using both explicit relevance judgments and large-scale log analysis of user behavior patterns. While variation in user behavior is correlated with variation in explicit relevance judgments the same query, there are many other factors, such as result entropy, result quality, and task that can also affect the variation in behavior. We characterize queries using a variety of features of the query, the results returned for the query, and people's interaction history with the query. Using these features we build predictive models to identify queries that can benefit from personalization.

[1]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[2]  Taher H. Haveliwala.  Topic-sensitive PageRank , 2002, IEEE Trans. Knowl. Data Eng..

[3]  Yong Yu,et al.  Identifying ambiguous queries in web search , 2007, WWW '07.

[4]  Jure Leskovec,et al.  Web projections: learning from contextual subgraphs of the web , 2007, WWW '07.

[5]  A. Spink,et al.  Web Search: Public Searching of the Web (Information Science and Knowledge Management) , 2005 .

[6]  ChengXiang Zhai,et al.  Implicit user modeling for personalized search , 2005, CIKM '05.

[7]  T. Saracevic,et al.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance , 2007, J. Assoc. Inf. Sci. Technol..

[8]  Wolfgang Nejdl,et al.  Using ODP metadata to personalize search , 2005, SIGIR '05.

[9]  Ryen W. White,et al.  WWW 2007 / Track: Browsers and User Interfaces Session: Personalization Investigating Behavioral Variability in Web Search , 2022 .

[10]  George Karypis,et al.  CLUTO - A Clustering Toolkit , 2002 .

[11]  Susan T. Dumais,et al.  Personalizing Search via Automated Analysis of Interests and Activities , 2005, SIGIR.

[12]  Amanda Spink,et al.  Web Search: Public Searching of the Web , 2011, Information Science and Knowledge Management.

[13]  W. Bruce Croft,et al.  Predicting query performance , 2002, SIGIR '02.

[14]  Elad Yom-Tov,et al.  What makes a query difficult? , 2006, SIGIR.

[15]  Susan T. Dumais,et al.  Potential for personalization , 2010, TCHI.

[16]  Daniel E. Rose,et al.  Understanding user goals in web search , 2004, WWW '04.

[17]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[18]  Raya Fidel,et al.  Users' perception of the performance of a filtering system , 1997, SIGIR '97.

[19]  References , 1971 .

[20]  W. Bruce Croft,et al.  Query performance prediction in web search environments , 2007, SIGIR.

[21]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR '00.

[22]  Ji-Rong Wen,et al.  WWW 2007 / Track: Search Session: Personalization A Largescale Evaluation and Analysis of Personalized Search Strategies ABSTRACT , 2022 .

[23]  Edward Cutrell,et al.  An eye tracking study of the effect of target rank on web search , 2007, CHI.

[24]  Oren Etzioni,et al.  On the Instability of Web Search Engines , 2000, RIAO.

[25]  Thorsten Joachims,et al.  Accurately interpreting clickthrough data as implicit feedback , 2005, SIGIR '05.

[26]  Zhenyu Liu,et al.  Automatic identification of user goals in Web search , 2005, WWW '05.

[27]  Tefko Saracevic,et al.  Relevance : A Review of the Literature and a Framework for Thinking on the Notion in Information Science . Part III : Behavior and Effects of Relevance , 1976 .