Top-k Consistency of Learning to Rank Methods

This paper is concerned with the consistency analysis on lis twise ranking methods. Among various ranking methods, the listwise methods ha ve competitive performances on benchmark datasets and are regarded as one of th e state-of-the-art approaches. Most listwise ranking methods manage to optimi ze ranking on the whole list (permutation) of objects, however, in practical applications such as information retrieval, correct ranking at the top k positions is much more important. This paper aims to analyze whether existing listwise rankin g methods are statistically consistent in the topk setting. For this purpose, we define a topk ranking framework, where the true loss (and thus the risks) are define d o the basis of topk subgroup of permutations. This framework can include the pe rmutation-level ranking framework proposed in previous work as a special cas e. Based on the new framework, we derive sufficient conditions for a listwis e ranking method to be consistent with the topk true loss, and show an effective way of modifying the surrogate loss functions in existing methods to satisfy the se conditions. Experimental results show that after the modification, the methods can work significantly better than their original versions, indicating the correc tn ss of our theoretical analysis.