ULTRA: An Unbiased Learning To Rank Algorithm Toolbox

Learning to rank system has become an important aspect of our daily life. However, the implicit user feedback that is used to train many learning to rank models is usually noisy and suffers from user bias (i.e., position bias). Thus, obtaining unbiased model using biased feedback has become an important research field for IR. Existing studies on unbiased learning to rank (ULTR) can be generalized into two families-algorithms that attain unbiasness with logged data, offline learning, and algorithms that achieve unbiasness by estimating unbiased parameters with real-time user interactions, namely online learning. While there exist many algorithms from both families, there lacks a unified way to compare and benchmark them. As a result, it can be challenging for researchers to choose the right technique for their problems or for people who are new to the field to learn and understand existing algorithms. To solve this problem, we introduced ULTRA, which is a flexible, extensible, and easily configure ULTR toolbox. Its key features include support for multiple ULTR algorithms with configurable hyper parameters, a variety of built-in click models that can be used separately to simulate clicks, different ranking model architectures and evaluation metrics, and simple learning to rank pipeline creation. In this paper, we discuss the general framework of ULTR, briefly describe the algorithms in ULTRA, detail the structure, and pipeline of the toolbox. We experimented on all the algorithms supported by ULTRA and showed that the toolbox performance is reasonable. Our toolbox is an important resource for researchers to conduct experiments on ULTR algorithms with different configurations as well as testing their own algorithms with the supported features.

[1]  Marc Najork,et al.  Position Bias Estimation for Unbiased Learning to Rank in Personal Search , 2018, WSDM.

[2]  W. Bruce Croft,et al.  Learning a Deep Listwise Context Model for Ranking Refinement , 2018, SIGIR.

[3]  Shaoping Ma,et al.  Constructing Click Models for Mobile Search , 2018, SIGIR.

[4]  Yang Wang,et al.  Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm , 2018, WWW.

[5]  Michael Bendersky,et al.  Addressing Trust Bias for Unbiased Learning-to-Rank , 2019, WWW.

[6]  Qingyao Ai,et al.  Unbiased Learning to Rank: Online or Offline? , 2020, ArXiv.

[7]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[8]  Thorsten Joachims,et al.  Accurately Interpreting Clickthrough Data as Implicit Feedback , 2017 .

[9]  M. de Rijke,et al.  Differentiable Unbiased Online Learning to Rank , 2018, CIKM.

[10]  Thorsten Joachims,et al.  Estimating Position Bias without Intrusive Interventions , 2018, WSDM.

[11]  Yiqun Liu,et al.  Incorporating vertical results into search click models , 2013, SIGIR.

[12]  Stephen E. Robertson,et al.  Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval , 1994, SIGIR '94.

[13]  Yiqun Liu,et al.  Unbiased Learning to Rank: Theory and Practice , 2018, ICTIR.

[14]  Thorsten Joachims,et al.  Interactively optimizing information retrieval systems as a dueling bandits problem , 2009, ICML '09.

[15]  Sebastian Bruch,et al.  Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks , 2018, ICTIR.

[16]  Filip Radlinski,et al.  Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search , 2007, TOIS.

[17]  Laura A. Granka,et al.  Accurately Interpreting Clickthrough Data as Implicit Feedback , 2017 .

[18]  Yanyan Lan,et al.  SetRank , 2020, Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.

[19]  M. de Rijke,et al.  Multileaved Comparisons for Fast Online Evaluation , 2014, CIKM.

[20]  Qiang Wu,et al.  McRank: Learning to Rank Using Multiple Classification and Gradient Boosting , 2007, NIPS.

[21]  Huazheng Wang,et al.  Efficient Exploration of Gradient Space for Online Learning to Rank , 2018, SIGIR.

[22]  Unbiased Learning-to-Rank with Biased Feedback , 2018, IJCAI.

[23]  Marc Najork,et al.  Learning to Rank with Selection Bias in Personal Search , 2016, SIGIR.

[24]  W. Bruce Croft,et al.  Correcting for Recency Bias in Job Recommendation , 2019, CIKM.

[25]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[26]  Samuel B. Williams,et al.  ASSOCIATION FOR COMPUTING MACHINERY , 2000 .

[27]  Qingyao Ai,et al.  Unbiased Learning to Rank , 2021, ACM Trans. Inf. Syst..

[28]  Mark T. Keane,et al.  Modeling Result-List Searching in the World Wide Web: The Role of Relevance Topologies and Trust Bias , 2006 .

[29]  Matthew Richardson,et al.  Predicting clicks: estimating the click-through rate for new ads , 2007, WWW '07.

[30]  Christopher J. C. Burges,et al.  From RankNet to LambdaRank to LambdaMART: An Overview , 2010 .

[31]  W. Bruce Croft,et al.  Unbiased Learning to Rank with Unbiased Propensity Estimation , 2018, SIGIR.

[32]  Thorsten Joachims,et al.  Unbiased Learning-to-Rank with Biased Feedback , 2016, WSDM.

[33]  Tie-Yan Liu,et al.  Learning to rank: from pairwise approach to listwise approach , 2007, ICML '07.

[34]  Marc Najork,et al.  Self-Attentive Document Interaction Networks for Permutation Equivariant Ranking , 2019, ArXiv.

[35]  Jun Xu,et al.  SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval , 2020, SIGIR.

[36]  Thorsten Joachims,et al.  A General Framework for Counterfactual Learning-to-Rank , 2018, SIGIR.

[37]  M. de Rijke,et al.  Multileave Gradient Descent for Fast Online Learning to Rank , 2016, WSDM.

[38]  Tie-Yan Liu,et al.  Learning to rank for information retrieval , 2009, SIGIR.