Ranking Aggregation Based on Belief Function Theory

The ranking aggregation problem is that to establishing a new aggregate ranking given a set of rankings of a finite set of items. This problem is met in various applications, such as the combination of user preferences, the combination of lists of documents retrieved by search engines and the combination of ranked gene lists. In the literature, the ranking aggregation problem has been solved as an optimization of some distance between the rankings overlooking the existence of a true ranking. In this thesis we address the ranking aggregation problem assuming the existence of a true ranking on the set of items: the goal is to estimate an unknown, true ranking given a set of input rankings provided by experts with different approximation quality. We propose a novel solution called Belief Ranking Estimator (BRE) that takes into account two aspects still unexplored in ranking combination: the approximation quality of the experts and for the first time the uncertainty related to each item position in the ranking. BRE estimates in an unsupervised way the true ranking given a set of rankings that are diverse quality estimations of the unknown true ranking. The uncertainty on the items's position in each ranking is modeled within the Belief Function Theory framework, that allows for the combination of subjective knowledge in a non Bayesian way. This innovative application of belief functions to rankings, allows us to encode different sources of a priori knowledge about the correctness of the ranking positions and also to weigh the reliability of the experts involved in the combination. We assessed the performance of our solution on synthetic and real data against state-of-the-art methods. The tests comprise the aggregation of total and partial rankings in different empirical settings aimed at representing the different quality of the input rankings with respect to the true ranking. The results show that BRE provides an effective solution when the input rankings are heterogeneous in terms of approximation quality with respect to the unknown true ranking.

[1]  Thierry Denoeux,et al.  Dempster-Shafer Reasoning in Large Partially Ordered Sets: Applications in Machine Learning , 2010, IUM.

[2]  Prakash P. Shenoy,et al.  On the plausibility transformation method for translating belief function models to probability models , 2006, Int. J. Approx. Reason..

[3]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[4]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[5]  Tie-Yan Liu,et al.  Learning to rank for information retrieval , 2009, SIGIR.

[6]  Philippe Smets,et al.  Decision making in the TBM: the necessity of the pignistic transformation , 2005, Int. J. Approx. Reason..

[7]  Vasyl Pihur,et al.  RankAggreg, an R package for weighted rank aggregation , 2009, BMC Bioinformatics.

[8]  Jie Ding,et al.  Integration of Ranked Lists via Cross Entropy Monte Carlo with Applications to mRNA and microRNA Studies , 2009, Biometrics.

[9]  John D. Lafferty,et al.  Conditional Models on the Ranking Poset , 2002, NIPS.

[10]  Thierry Denoeux,et al.  ECM: An evidential version of the fuzzy c , 2008, Pattern Recognit..

[11]  R. Lo Cigno,et al.  On some fundamental properties of P2P push/pull protocols , 2008, 2008 Second International Conference on Communications and Electronics.

[12]  Enrico Blanzieri,et al.  About Neighborhood Counting Measure Metric and Minimum Risk Metric , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Thierry Denoeux,et al.  An evidence-theoretic k-NN rule with parameter optimization , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[14]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR '00.

[15]  Shili Lin,et al.  Rank aggregation methods , 2010 .

[16]  Thierry Denoeux,et al.  EVCLUS: evidential clustering of proximity data , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[17]  Noureddine Zerhouni,et al.  E2GK: Evidential Evolving Gustafsson-Kessel Algorithm for Data Streams Partitioning Using Belief Functions , 2011, ECSQARU.

[18]  D. Critchlow.  Metric Methods for Analyzing Partially Ranked Data , 1986 .

[19]  Martin Reczko,et al.  Lost in translation: an assessment and perspective for computational microRNA target identification , 2009, Bioinform..

[20]  Vasyl Pihur,et al.  Weighted rank aggregation of cluster validation measures: a Monte Carlo cross-entropy approach , 2007, Bioinform..

[21]  Jerome L. Myers,et al.  Research Design and Statistical Analysis , 1991 .

[22]  Dan Roth,et al.  An Unsupervised Learning Algorithm for Rank Aggregation , 2007, ECML.

[23]  R. Duncan Luce,et al.  Individual Choice Behavior , 1959 .

[24]  R. Graham,et al.  Spearman's Footrule as a Measure of Disarray , 1977 .

[25]  Philippe Smets,et al.  Classification Using Belief Functions: Relationship Between Case-Based and Model-Based Approaches , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[26]  Tao Qin,et al.  A New Probabilistic Model for Rank Aggregation , 2010, NIPS.

[27]  R. Yager.  On the dempster-shafer framework and new combination rules , 1987, Inf. Sci..

[28]  A. Dempster.  Upper and Lower Probabilities Generated by a Random Closed Interval , 1968 .

[29]  Ronald R. Yager,et al.  Decision Making Under Dempster-Shafer Uncertainties , 1992, Classic Works of the Dempster-Shafer Theory of Belief Functions.

[30]  Philippe Smets,et al.  The Transferable Belief Model , 1991, Artif. Intell..

[31]  Moni Naor,et al.  Rank aggregation methods for the Web , 2001, WWW '01.

[32]  L. V. Jones,et al.  The Rational Origin for Measuring Subjective Values , 1957 .

[33]  Enrico Blanzieri,et al.  Ranking Aggregation Based on Belief Function , 2012, IPMU.

[34]  Ralf Herbrich,et al.  Large margin rank boundaries for ordinal regression , 2000 .

[35]  Johan Schubert.  Clustering decomposed belief functions using generalized weights of conflict , 2008, Int. J. Approx. Reason..

[36]  C. L. Mallows.  NON-NULL RANKING MODELS. I , 1957 .

[37]  L. Thurstone.  Rank order as a psycho-physical method. , 1931 .

[38]  Tao Qin,et al.  LETOR: Benchmark Dataset for Research on Learning to Rank for Information Retrieval , 2007 .

[39]  Malik Magdon-Ismail,et al.  The Impact of Ranker Quality on Rank Aggregation Algorithms: Information vs. Robustness , 2006, 22nd International Conference on Data Engineering Workshops (ICDEW'06).

[40]  Thierry Denoeux,et al.  Fault diagnosis in railway track circuits using Dempster-Shafer classifier fusion , 2010, Eng. Appl. Artif. Intell..

[41]  Thierry Denoeux,et al.  An Evidence-Theoretic k-Nearest Neighbor Rule for Multi-label Classification , 2009, SUM.

[42]  Henri Prade,et al.  Representation and combination of uncertainty with belief functions and possibility measures , 1988, Comput. Intell..

[43]  Thierry Denoeux.  The cautious rule of combination for belief functions and some extensions , 2006, 2006 9th International Conference on Information Fusion.

[44]  E. Blanzieri,et al.  Unsupervised Learning of True Ranking Estimators using the Belief Function Framework , 2011 .

[45]  Ronald Fagin,et al.  Efficient similarity search and classification via rank aggregation , 2003, SIGMOD '03.

[46]  R. Plackett.  The Analysis of Permutations , 1975 .

[47]  Tao Qin,et al.  LETOR: A benchmark collection for research on learning to rank for information retrieval , 2010, Information Retrieval.

[48]  Glenn Shafer,et al.  Belief-Function Formulas for Audit Risk , 2008, Classic Works of the Dempster-Shafer Theory of Belief Functions.

[49]  H. Young.  Condorcet's Theory of Voting , 1988, American Political Science Review.

[50]  R. Forthofer,et al.  Rank Correlation Methods , 1981 .

[51]  Ronald Fagin,et al.  Comparing top k lists , 2003, SODA '03.

[52]  Dan Roth,et al.  Unsupervised rank aggregation with distance-based models , 2008, ICML '08.