Measuring and Predicting Search Engine Users’ Satisfaction

Search satisfaction is defined as the fulfillment of a user’s information need. Characterizing and predicting the satisfaction of search engine users is vital for improving ranking models, increasing user retention rates, and growing market share. This article provides an overview of the research areas related to user satisfaction. First, we show that whenever users choose to defect from one search engine to another they do so mostly due to dissatisfaction with the search results. We also describe several search engine switching prediction methods, which could help search engines retain more users. Second, we discuss research on the difference between good and bad abandonment, which shows that in approximately 30% of all abandoned searches the users are in fact satisfied with the results. Third, we catalog techniques to determine queries and groups of queries that are underperforming in terms of user satisfaction. This can help improve search engines by developing specialized rankers for these query patterns. Fourth, we detail how task difficulty affects user behavior and how task difficulty can be predicted. Fifth, we characterize satisfaction and we compare major satisfaction prediction algorithms.

[1]  Ryen W. White,et al.  Personalized models of search satisfaction , 2013, CIKM.

[2]  W. Bruce Croft,et al.  Predicting query performance , 2002, SIGIR '02.

[3]  Rolph E. Anderson,et al.  E‐satisfaction and e‐loyalty: A contingency framework , 2003 .

[4]  Ruth N. Bolton,et al.  A Dynamic Model of the Duration of the Customer's Relationship with a Continuous Service Provider: The Role of Satisfaction , 1994 .

[5]  Nicholas J. Belkin,et al.  Predicting Search Task Difficulty at Different Search Stages , 2014, CIKM.

[6]  Ryen W. White,et al.  Effects of search success on search engine re-use , 2011, CIKM '11.

[7]  Yang Song,et al.  Modeling action-level satisfaction for search task satisfaction prediction , 2014, SIGIR.

[8]  Thorsten Joachims,et al.  Accurately interpreting clickthrough data as implicit feedback , 2005, SIGIR '05.

[9]  Anne Aula,et al.  How does search behavior change as search becomes more difficult? , 2010, CHI.

[10]  P. Wilton,et al.  Models of Consumer Satisfaction Formation : An Extension , 1988 .

[11]  Ronald D. Cassell,et al.  Churchill , 2001 .

[12]  João Leite,et al.  Playing with Rules , 2009, DALT.

[13]  Ricardo Baeza-Yates,et al.  Improved query difficulty prediction for the web , 2008, CIKM '08.

[14]  Ryen W. White,et al.  Why searchers switch: understanding and predicting engine switching rationales , 2011, SIGIR.

[15]  Mark Sanderson,et al.  The relationship between IR effectiveness measures and user satisfaction , 2007, SIGIR.

[16]  Nicholas J. Belkin,et al.  Exploring and predicting search task difficulty , 2012, CIKM '12.

[17]  Ryen W. White,et al.  Mining for insights in the search engine query stream , 2012, WWW.

[18]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[19]  Iadh Ounis,et al.  Inferring Query Performance Using Pre-retrieval Predictors , 2004, SPIRE.

[20]  James Allan,et al.  Predicting searcher frustration , 2010, SIGIR.

[21]  Steve Fox,et al.  Evaluating implicit measures to improve web search , 2005, TOIS.

[22]  Peter Kulchyski.  and , 2015 .

[23]  Ryen W. White,et al.  Leaving so soon?: understanding and predicting web search abandonment rationales , 2012, CIKM.

[24]  Dan Morris,et al.  Investigating the querying and browsing behavior of advanced search engine users , 2007, SIGIR.

[25]  共立出版株式会社.  コンピュータ・サイエンス : ACM computing surveys , 1978 .

[26]  Ryen W. White,et al.  Struggling or exploring?: disambiguating long search sessions , 2014, WSDM.

[27]  Jaime Arguello.  Predicting Search Task Difficulty , 2014, ECIR.

[28]  Ryen W. White,et al.  Modeling dwell time to predict click-level satisfaction , 2014, WSDM.

[29]  Yang Song,et al.  Context-aware web search abandonment prediction , 2014, SIGIR.

[30]  Vikas Mittal.  Customer‐Satisfaction Research , 2010 .

[31]  B Y Tim.  Playing by the Rules , 2013 .

[32]  Dmitry Lagun,et al.  Search engine switching detection based on user personal preferences and behavior patterns , 2013, SIGIR.

[33]  Ryen W. White,et al.  Stream prediction using a generative model based on frequent episodes in event sequences , 2008, KDD.

[34]  T. Hennig-Thurau,et al.  The impact of customer satisfaction and relationship quality on customer retention: A critical reassessment and model development , 1997 .

[35]  Ahmed Hassan Awadallah,et al.  A semi-supervised approach to modeling web search satisfaction , 2012, SIGIR '12.

[36]  Ryen W. White,et al.  Modeling Long-Term Search Engine Usage , 2010, UMAP.

[37]  Thomas O. Jones,et al.  Why Satisfied Customers Defect , 1996 .

[38]  J. Jacoby,et al.  Consumer Behavior , 2024 .

[39]  Ryen W. White,et al.  Understanding and Predicting Graded Search Satisfaction , 2015, WSDM.

[40]  Iadh Ounis,et al.  Query performance prediction , 2006, Inf. Syst..

[41]  Ryen W. White,et al.  Toward self-correcting search engines: using underperforming queries to improve search , 2013, SIGIR.

[42]  Ryen W. White,et al.  Defection detection: predicting search engine switching , 2008, WWW.

[43]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[44]  Bernard J. Jansen,et al.  Almighty Twitter, what are people asking for? , 2012, ASIST.

[45]  Aleksandr Chuklin,et al.  Potential good abandonment prediction , 2012, WWW.

[46]  S. Seyedaliakbar,et al.  An investigation into the determinants of customer satisfaction , 2016 .

[47]  Lydia B. Chilton,et al.  Addressing people's information needs directly in a web search result page , 2011, WWW.

[48]  David G. Stork,et al.  Pattern Classification , 1973 .

[49]  Nick Craswell,et al.  Beyond clicks: query reformulation as a predictor of search satisfaction , 2013, CIKM.

[50]  Jacek Gwizdka,et al.  Predicting task difficulty for different task types , 2010, ASIST.

[51]  Nicholas J. Belkin,et al.  Task difficulty and domain knowledge effects on information search behaviors , 2012, ASIST.

[52]  Yun-Fang Juan,et al.  An analysis of search engine switching behavior using click streams , 2005, WWW '05.

[53]  Ahmed Hassan Awadallah,et al.  Beyond DCG: user behavior as a predictor of a successful search , 2010, WSDM '10.

[54]  Jane Li,et al.  Good abandonment in mobile and PC internet search , 2009, SIGIR.

[55]  Rahul Telang,et al.  Competition between Internet search engines , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[56]  Sofia Stamou,et al.  Interpreting User Inactivity on Search Results , 2010, ECIR.

[57]  Jeonghyun Kim,et al.  Task difficulty as a predictor and indicator of web searching interaction , 2006, CHI Extended Abstracts.

[58]  Ryen W. White,et al.  WWW 2007 / Track: Browsers and User Interfaces Session: Personalization Investigating Behavioral Variability in Web Search , 2022 .

[59]  Ryen W. White,et al.  Characterizing and predicting search engine switching behavior , 2009, CIKM.

[60]  S. Beatty,et al.  External Search Effort: An Investigation across Several Product Categories , 1987 .

[61]  Aleksandr Chuklin,et al.  Good abandonments in factoid queries , 2012, WWW.

[62]  Yang Song,et al.  A task level metric for measuring web search satisfaction and its application on improving relevance estimation , 2011, CIKM '11.

[63]  Ryen W. White,et al.  No clicks, no problem: using cursor movements to understand and improve search , 2011, CHI.

[64]  Elad Yom-Tov,et al.  What makes a query difficult? , 2006, SIGIR.

[65]  Jacek Gwizdka,et al.  Can search systems detect users' task difficulty?: some behavioral signals , 2010, SIGIR '10.

[66]  Ryen W. White,et al.  Predicting query performance using query, result, and user interaction features , 2010, RIAO.

[67]  Ryen W. White,et al.  Enhancing web search by promoting multiple search engine use , 2008, SIGIR '08.

[68]  Scott B. Huffman,et al.  How well does result relevance predict session satisfaction? , 2007, SIGIR.