Tailoring click models to user goals

Click models provide a principled way of understanding user interaction with web search results in a query session and a statistical tool for leveraging search engine click logs to analyze and improve user experience. An important component in all existing click models is the user behavior assumption -- how users scan, examine and click web documents listed in the result page. Usually the average user behavior pattern is summarized in a small set of global parameters. Can we fit multiple models with different user behavior parameters on a click data set? A previous study showed that the mixture modeling approach did not lead to better performance despite extra computational cost. In this paper, we present how to tailor click models to user goals in web search through query term classification. We demonstrate that better predicative power could be achieved by fitting two click models for navigational queries and informational queries respectively, as evidenced by the likelihood and perplexity evaluation results on a subset of the MSN 2006 RFP data which consists of 121,179 distinct query terms and over 2.8 million query sessions. We also propose search relevance score (SRS) as a flexible evaluation metric of search engine performance. This metric can be derived as summary statistics under any click model, and is applicable to a single query session, a particular query term and the search engine overall.