Investigating Result Usefulness in Mobile Search

The existing evaluation approaches for search engines usually measure and estimate the utility or usefulness of search results by either the explicit relevance annotations from external assessors or implicit behavior signals from users. Because the mobile search is different from the desktop search in terms of the search tasks and the presentation styles of SERPs, whether the approaches originated from the desktop settings are still valid in the mobile scenario needs further investigation. To address this problem, we conduct a laboratory user study to record users’ search behaviors and collect their usefulness feedbacks for search results when using mobile devices. By analyzing the collected data, we investigate and characterize how the relevance, as well as the ranking position and presentation style of a result, affects its user-perceived usefulness level. A moderating effect of presentation style on the correlation between relevance and usefulness as well as a position bias affecting the usefulness in the initial viewport are identified. By correlating result-level usefulness feedbacks and relevance annotations with query-level satisfaction, we confirm the findings that usefulness feedbacks can better reflect user satisfaction than relevance annotations in mobile search. We also study the relationship between users’ usefulness feedbacks and their implicit search behavior, showing that the viewport features can be used to estimate usefulness when click signals are absent. Our study highlights the difference between desktop and mobile search and sheds light on developing a more user-centric evaluation method for mobile search.