Investigating the Reliability of Click Models

Click models aim to extract accurate relevance feedback from the noisy and biased user clicks. Previous work focuses on reducing the systematic bias between click and relevance but few studies have examined the reliability and precision of click models' relevance estimation. So in this study, we propose to investigate the reliability of relevance estimation derived by click models. Instead of getting a point estimate of relevance, a variational Bayesian method is used to infer the posterior distribution of relevance parameters. Based on the posterior distribution, we define measures for the reliability of pointwise and pairwise relevance estimation. With experiments on both real and synthetic query logs, we show that: 1) the proposed method effectively captures the uncertainty in relevance estimation; 2) the reliability of click models' relevance estimation is affected by the size of training data, the average ranking position of documents, and the ranking strategy of search engines.