Ranking forests

The present paper examines how the aggregation and feature randomization principles underlying the algorithm RANDOM FOREST (Breiman, 2001) can be adapted to bipartite ranking. The approach taken here is based on nonparametric scoring and ROC curve optimization in the sense of the AUC criterion. In this problem, aggregation is used to increase the performance of scoring rules produced by ranking trees, as those developed in Cleemencon and Vayatis (2009c). The present work describes the principles for building median scoring rules based on concepts from rank aggregation. Consistency results are derived for these aggregated scoring rules and an algorithm called RANKING FOREST is presented. Furthermore, various strategies for feature randomization are explored through a series of numerical experiments on artificial data sets.

[1]  Jeff A. Bilmes,et al.  Consensus ranking under the exponential model , 2007, UAI.

[2]  Alain Rakotomamonjy,et al.  Optimizing Area Under Roc Curve with SVMs , 2004, ROCAI.

[3]  James P. Egan,et al.  Signal detection theory and ROC analysis , 1975 .

[4]  Stéphan Clémençon,et al.  Overlaying classifiers: a practical approach for optimal ranking , 2008, NIPS.

[5]  John D. Lafferty,et al.  Conditional Models on the Ranking Poset , 2002, NIPS.

[6]  R. Tibshirani,et al.  Generalized additive models for medical research , 1986, Statistical methods in medical research.

[7]  Walid G. Aref,et al.  Joining Ranked Inputs in Practice , 2002, VLDB.

[8]  Joseph S. Verducci,et al.  Probability Models and Statistical Analyses for Ranking Data , 1992 .

[9]  Stéphan Clémençon,et al.  Bagging Ranking Trees , 2009, 2009 International Conference on Machine Learning and Applications.

[10]  S. Shapiro,et al.  Mathematics without Numbers , 1993 .

[11]  J. Hilbe Logistic Regression Models , 2009 .

[12]  Yoram Singer,et al.  Learning to Order Things , 1997, NIPS.

[13]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[14]  Dan Roth,et al.  Generalization Bounds for the Area Under the ROC Curve , 2005, J. Mach. Learn. Res..

[15]  James C. Spall,et al.  Introduction to Stochastic Search and Optimization. Estimation, Simulation, and Control (Spall, J.C. , 2007 .

[16]  Ronald Fagin,et al.  Comparing Partial Rankings , 2006, SIAM J. Discret. Math..

[17]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[18]  Stéphan Clémençon,et al.  Tree-structured ranking rules and approximation of the optimal ROC curve , 2008 .

[19]  Ronald Fagin,et al.  Comparing and aggregating rankings with ties , 2004, PODS '04.

[20]  Stéphan Clémençon,et al.  AUC optimization and the two-sample problem , 2009, NIPS.

[21]  Elena Deza,et al.  Encyclopedia of Distances , 2014 .

[22]  Pierre Bernard,et al.  Lectures on Probability Theory and Statistics: Ecole d'Ete de Probabilites de Saint-Flour XXVI - 1996 , 1997 .

[23]  E. Giné,et al.  Decoupling: From Dependence to Independence , 1998 .

[24]  Olivier Hudry,et al.  NP-hardness results for the aggregation of linear orders into median orders , 2008, Ann. Oper. Res..

[25]  J. Friedman,et al.  On bagging and nonlinear estimation , 2007 .

[26]  M. Kendall The treatment of ties in ranking problems. , 1945, Biometrika.

[27]  Jean-Philippe Vert,et al.  Consistency of Random Forests , 2014, 1405.2881.

[28]  T. Salakoski,et al.  Learning to Rank with Pairwise Regularized Least-Squares , 2007 .

[29]  Marina Meila,et al.  Tractable Search for Learning Exponential Models of Rankings , 2009, AISTATS.

[30]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[31]  Bernard Monjardet,et al.  The median procedure in cluster analysis and social choice theory , 1981, Math. Soc. Sci..

[32]  Eric Horvitz,et al.  Social Choice Theory and Recommender Systems: Analysis of the Axiomatic Foundations of Collaborative Filtering , 2000, AAAI/IAAI.

[33]  Yoshiko Wakabayashi The Complexity of Computing Medians of Relations , 1998 .

[34]  G. Lugosi,et al.  Ranking and empirical minimization of U-statistics , 2006, math/0603123.

[35]  Janis E. Johnston,et al.  Permutation methods , 2001 .

[36]  N. Vayatis,et al.  Overlaying Classifiers: A Practical Approach to Optimal Scoring , 2010 .

[37]  P. Fishburn The Theory Of Social Choice , 1973 .

[38]  David Fernández-Baca,et al.  Computing distances between partial rankings , 2009, Inf. Process. Lett..

[39]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[40]  Stéphan Clémençon,et al.  Tree-Based Ranking Methods , 2009, IEEE Transactions on Information Theory.

[41]  Michael R. Fellows,et al.  Computing Kemeny rankings, parameterized by the average K-T distance , 2008 .

[42]  A. V. D. Vaart,et al.  Lectures on probability theory and statistics , 2002 .

[43]  Stéphan Clémençon,et al.  Adaptive partitioning schemes for bipartite ranking , 2011, Machine Learning.

[44]  Nicolas Vayatis,et al.  R-implementation of the TreeRank algorithm , 2009 .

[45]  Peter A. Flach,et al.  Learning Decision Trees Using the Area Under the ROC Curve , 2002, ICML.

[46]  R. Dudley,et al.  Uniform Central Limit Theorems: Notation Index , 2014 .

[47]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[48]  Robert P. Sheridan,et al.  Random Forest: A Classification and Regression Tool for Compound Classification and QSAR Modeling , 2003, J. Chem. Inf. Comput. Sci..

[49]  Yali Amit,et al.  Shape Quantization and Recognition with Randomized Trees , 1997, Neural Computation.

[50]  Sylvain Arlot Model selection by resampling penalization , 2007, 0906.3124.

[51]  P. Diaconis A Generalization of Spectral Analysis with Application to Ranked Data , 1989 .

[52]  Ramón Díaz-Uriarte,et al.  Gene selection and classification of microarray data using random forest , 2006, BMC Bioinformatics.

[53]  Moni Naor,et al.  Rank aggregation methods for the Web , 2001, WWW '01.

[54]  A. O. Houcine On hyperbolic groups , 2006 .

[55]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[56]  Irène Charon,et al.  Lamarckian genetic algorithmsapplied to the aggregation of preferences , 1998, Ann. Oper. Res..

[57]  Eyke Hüllermeier,et al.  Label ranking by learning pairwise preferences , 2008, Artif. Intell..

[58]  Rafael Martí,et al.  Intensification and diversification with elite tabu search solutions for the linear ordering problem , 1999, Comput. Oper. Res..

[59]  James C. Spall,et al.  Introduction to stochastic search and optimization - estimation, simulation, and control , 2003, Wiley-Interscience series in discrete mathematics and optimization.

[60]  Stéphan Clémençon,et al.  Ranking the Best Instances , 2006, J. Mach. Learn. Res..

[61]  Yves Grandvalet,et al.  Bagging Equalizes Influence , 2004, Machine Learning.

[62]  Luc Devroye,et al.  Consistency of Random Forests and Other Averaging Classifiers , 2008, J. Mach. Learn. Res..

[63]  Gábor Lugosi,et al.  Ranking and Scoring Using Empirical Risk Minimization , 2005, COLT.

[64]  Tim Hesterberg,et al.  Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control , 2004, Technometrics.