In our participation to the TREC 2020 Fair Ranking Track, as Naver Labs Europe, we focused on the re-ranking task and we investigated the performance of a controller as a way to minimize unfairness over time, with minimal loss of utility. We used a two-step approach, based on (1) a relevance probability estimator, and (2) a controller that aims to bring the actual exposure close to the target exposure. This paper describes the components we designed in more detail. It contains a comparison of the performance of the controller to a baseline, which consists of a Plackett-Luce sampler. It also analyses the effect of the quality of the estimated relevance probabilities (closeness to the true binary relevance values) on the controller performance.
[1]
Bhaskar Mitra,et al.
Evaluating Stochastic Rankings with Expected Exposure
,
2020,
CIKM.
[2]
Bhaskar Mitra,et al.
A Dual Embedding Space Model for Document Ranking
,
2016,
ArXiv.
[3]
Yili Hong,et al.
On computing the distribution function for the Poisson binomial distribution
,
2013,
Comput. Stat. Data Anal..
[4]
Rich Caruana,et al.
Predicting good probabilities with supervised learning
,
2005,
ICML.
[5]
Karen Spärck Jones,et al.
A probabilistic model of information retrieval: development and comparative experiments - Part 1
,
2000,
Inf. Process. Manag..