Investigating Fine-Grained Usefulness Perception Process in Mobile Search

With the development and popularization of smartphones, search on mobile devices has become more and more popular in recent years. Existing research found that users’ search interaction patterns in the mobile environment are different from those in the desktop environment. As we know, there are a number of vertical results and richly informative snippets in the ranked lists of mobile search engines. Users can perceive useful information from both the snippet and the landing page of a result. Therefore, we consider that it is necessary to investigate how users interact with mobile search engine result pages and their fine-grained usefulness perception processes. In this paper, we collected fine-grained usefulness annotations for mobile search results in a user study dataset. With the user behavior information in the dataset, we investigate the patterns of users’ examination and click behavior and propose a user model for the fine-grained usefulness perception process in mobile search. Our research sheds light on improving user models in mobile search evaluation metrics and other mobile search-related applications.

[1]  Tamás D. Gedeon,et al.  Understanding eye movements on mobile devices for better presentation of search results , 2015, J. Assoc. Inf. Sci. Technol..

[2]  Peter Bailey,et al.  User Variability and IR System Evaluation , 2015, SIGIR.

[3]  Eugene Agichtein,et al.  Mining touch interaction data on mobile devices to predict web search result relevance , 2013, SIGIR.

[4]  Alistair Moffat,et al.  Rank-biased precision for measurement of retrieval effectiveness , 2008, TOIS.

[5]  Yiqun Liu,et al.  Investigating Examination Behavior in Mobile Search , 2020, WSDM.

[6]  Tamás D. Gedeon,et al.  Eye‐tracking analysis of user behavior and performance in web search on large and small screens , 2015, J. Assoc. Inf. Sci. Technol..

[7]  Chih-Hung Hsieh,et al.  Towards better measurement of attention and satisfaction in mobile search , 2014, SIGIR.

[8]  Vidhya Navalpakkam,et al.  Understanding Mobile Searcher Attention with Rich Ad Formats , 2016, CIKM.

[9]  Shaoping Ma,et al.  Constructing Click Models for Mobile Search , 2018, SIGIR.

[10]  Mohan S. Kankanhalli,et al.  Attentive Long Short-Term Preference Modeling for Personalized Product Search , 2018, ACM Trans. Inf. Syst..

[11]  Fan Zhang,et al.  Evaluating Mobile Search with Height-Biased Gain , 2017, SIGIR.

[12]  Alistair Moffat,et al.  Users versus models: what observation tells us about effectiveness metrics , 2013, CIKM.

[13]  Fan Zhang,et al.  Evaluating Web Search with a Bejeweled Player Model , 2017, SIGIR.

[14]  Yiqun Liu,et al.  Constructing Click Model for Mobile Search with Viewport Time , 2019, ACM Trans. Inf. Syst..

[15]  KimJaewon,et al.  Eye-tracking analysis of user behavior and performance in web search on large and small screens , 2015 .

[16]  Yang Song,et al.  Exploring and exploiting user search behavior on mobile and tablet devices to improve search relevance , 2013, WWW '13.

[17]  Nikolaos K. Tselios,et al.  Does size matter?: investigating the impact of mobile phone screen size on users' perceived usability, effectiveness and efficiency. , 2013, MobileHCI '13.

[18]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR Forum.

[19]  Yiqun Liu,et al.  Investigating Passage-level Relevance and Its Role in Document-level Relevance Judgment , 2019, SIGIR.

[20]  Yiqun Liu,et al.  A Large-Scale Study of Mobile Search Examination Behavior , 2018, SIGIR.

[21]  Olivier Chapelle,et al.  Expected reciprocal rank for graded relevance , 2009, CIKM.

[22]  Andrew Trotman,et al.  Sound and complete relevance assessment for XML retrieval , 2008, TOIS.