Learning to rank for information retrieval

This tutorial is concerned with a comprehensive introduction to the research area of learning to rank for information retrieval. In the first part of the tutorial, we will introduce three major approaches to learning to rank, i.e., the pointwise, pairwise, and listwise approaches, analyze the relationship between the loss functions used in these approaches and the widely-used IR evaluation measures, evaluate the performance of these approaches on the LETOR benchmark datasets, and demonstrate how to use these approaches to solve real ranking applications. In the second part of the tutorial, we will discuss some advanced topics regarding learning to rank, such as relational ranking, diverse ranking, semi-supervised ranking, transfer ranking, query-dependent ranking, and training data preprocessing. In the third part, we will briefly mention the recent advances on statistical learning theory for ranking, which explain the generalization ability and statistical consistency of different ranking methods. In the last part, we will conclude the tutorial and show several future research directions.

[1]  Massih-Reza Amini,et al.  A boosting algorithm for learning bipartite ranking functions with partially labeled data , 2008, SIGIR '08.

[2]  S. Rajaram,et al.  Generalization Bounds for k-Partite Ranking , 2005 .

[3]  Aravind K. Joshi,et al.  Ranking and Reranking with Perceptron , 2005, Machine Learning.

[4]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR '00.

[5]  Yong Yu,et al.  Learning to rank with ties , 2008, SIGIR '08.

[6]  Marc Sapoval,et al.  Advertisement , 2003, Frontiers in Neuroendocrinology.

[7]  Tao Qin,et al.  A general approximation framework for direct optimization of information retrieval measures , 2010, Information Retrieval.

[8]  Tao Qin,et al.  Learning to rank relational objects and its application to web search , 2008, WWW.

[9]  Stéphan Clémençon,et al.  Ranking the Best Instances , 2006, J. Mach. Learn. Res..

[10]  Bo Pang,et al.  Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales , 2005, ACL.

[11]  Norbert Fuhr,et al.  Optimum polynomial retrieval functions based on the probability ranking principle , 1989, TOIS.

[12]  Gerhard Widmer,et al.  Prediction of Ordinal Classes Using Regression Trees , 2001, Fundam. Informaticae.

[13]  Tao Qin,et al.  Learning to Search Web Pages with Query-Level Loss Functions , 2006 .

[14]  Dan Roth,et al.  Generalization Bounds for the Area Under the ROC Curve , 2005, J. Mach. Learn. Res..

[15]  Farzin Maghoul,et al.  Query clustering using click-through graph , 2009, WWW '09.

[16]  Tao Qin,et al.  Feature selection for ranking , 2007, SIGIR.

[17]  Pável Calado,et al.  A combined component approach for finding collection-adapted ranking functions based on genetic programming , 2007, SIGIR.

[18]  Benjamin Piwowarski,et al.  A user browsing model to predict search engine click data from past observations. , 2008, SIGIR '08.

[19]  Yi-Hsuan Yang,et al.  Video search reranking via online ordinal reranking , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[20]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[21]  John Guiver,et al.  Bayesian inference for Plackett-Luce ranking models , 2009, ICML '09.

[22]  Yanyan Lan,et al.  A Unified View of Loss Functions in Learning to Rank , 2009 .

[23]  Tao Qin,et al.  Ranking with query-dependent loss for web search , 2010, WSDM '10.

[24]  Thomas Hofmann,et al.  Support vector machine learning for interdependent and structured output spaces , 2004, ICML.

[25]  Changhu Wang,et al.  Learning query-biased web page summarization , 2007, CIKM '07.

[26]  Tao Qin,et al.  Global Ranking Using Continuous Conditional Random Fields , 2008, NIPS.

[27]  M. Kendall A NEW MEASURE OF RANK CORRELATION , 1938 .

[28]  Susan T. Dumais,et al.  Evaluating implicit measures to improve the search experiences , 2003 .

[29]  Naonori Ueda,et al.  Generalization error of ensemble estimators , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[30]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[31]  Tao Qin,et al.  Supervised rank aggregation , 2007, WWW '07.

[32]  Thorsten Joachims,et al.  Predicting diverse subsets using structural SVMs , 2008, ICML '08.

[33]  Jaime G. Carbonell,et al.  Optimizing estimated loss reduction for active sampling in rank learning , 2008, ICML '08.

[34]  Shivani Agarwal Generalization Bounds for Some Ordinal Regression Algorithms , 2008, ALT.

[35]  Yoram Singer,et al.  Learning to Order Things , 1997, NIPS.

[36]  Edward Y. Chang,et al.  Parallelizing Support Vector Machines on Distributed Computers , 2007, NIPS.

[37]  Thomas Hofmann,et al.  Learning to Rank with Nonsmooth Cost Functions , 2006, NIPS.

[38]  W. Bruce Croft,et al.  A Markov random field model for term dependencies , 2005, SIGIR '05.

[39]  Martin Szummer,et al.  A Decision Theoretic Framework for Ranking using Implicit Feedback , 2008 .

[40]  Filip Radlinski,et al.  Learning diverse rankings with multi-armed bandits , 2008, ICML '08.

[41]  Thomas Hofmann,et al.  Large Margin Methods for Structured and Interdependent Output Variables , 2005, J. Mach. Learn. Res..

[42]  Rong Jin,et al.  Semi-Supervised Learning by Mixed Label Propagation , 2007, AAAI.

[43]  Christopher D. Manning,et al.  Introduction to Information Retrieval , 2010, J. Assoc. Inf. Sci. Technol..

[44]  S. Robertson The probability ranking principle in IR , 1997 .

[45]  Tong Zhang,et al.  Subset Ranking Using Regression , 2006, COLT.

[46]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[47]  R. Duncan Luce,et al.  Individual Choice Behavior , 1959 .

[48]  Tie-Yan Liu,et al.  Future directions in learning to rank , 2010, Yahoo! Learning to Rank Challenge.

[49]  Tao Qin,et al.  A New Probabilistic Model for Rank Aggregation , 2010, NIPS.

[50]  Koby Crammer,et al.  Pranking with Ranking , 2001, NIPS.

[51]  W. Bruce Croft,et al.  A language modeling approach to information retrieval , 1998, SIGIR '98.

[52]  Filip Radlinski,et al.  A support vector method for optimizing average precision , 2007, SIGIR.

[53]  Tao Qin,et al.  Query-level stability and generalization in learning to rank , 2008, ICML '08.

[54]  R. Forthofer,et al.  Rank Correlation Methods , 1981 .

[55]  Hang Li,et al.  Cost-Sensitive Learning of SVM for Ranking , 2006, ECML.

[56]  Tong Zhang Statistical behavior and consistency of classification methods based on convex risk minimization , 2003 .

[57]  Wei Chu,et al.  Gaussian Processes for Ordinal Regression , 2005, J. Mach. Learn. Res..

[58]  O. Chapelle Large margin optimization of ranking measures , 2007 .

[59]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2004 .

[60]  Hang Li,et al.  AdaRank: a boosting algorithm for information retrieval , 2007, SIGIR.

[61]  Dan Roth,et al.  Unsupervised rank aggregation with distance-based models , 2008, ICML '08.

[62]  Kevin Duh,et al.  Learning to rank with partially-labeled data , 2008, SIGIR '08.

[63]  Ji-Rong Wen,et al.  Query clustering using user logs , 2002, TOIS.

[64]  Eric Brill,et al.  Learning effective ranking functions for newsgroup search , 2004, SIGIR '04.

[65]  Yiming Yang,et al.  A Loss Function Analysis for Classification Methods in Text Categorization , 2003, ICML.

[66]  Edward F. Harrington,et al.  Online Ranking/Collaborative Filtering Using the Perceptron Algorithm , 2003, ICML.

[67]  Klaus Obermayer,et al.  Support vector learning for ordinal regression , 1999 .

[68]  Edward A. Fox,et al.  Ranking function optimization for effective Web search by genetic programming: an empirical study , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[69]  Javed A. Aslam,et al.  Models for metasearch , 2001, SIGIR '01.

[70]  C. L. Mallows NON-NULL RANKING MODELS. I , 1957 .

[71]  Hongyuan Zha,et al.  Query-level learning to rank using isotonic regression , 2008, 2008 46th Annual Allerton Conference on Communication, Control, and Computing.

[72]  CHENGXIANG ZHAI,et al.  A study of smoothing methods for language models applied to information retrieval , 2004, TOIS.

[73]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[74]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[75]  David M. Pennock,et al.  Mining the peanut gallery: opinion extraction and semantic classification of product reviews , 2003, WWW '03.

[76]  Tao Qin,et al.  Robust sparse rank learning for non-smooth ranking measures , 2009, SIGIR.

[77]  Emine Yilmaz,et al.  Estimating average precision with incomplete and imperfect judgments , 2006, CIKM '06.

[78]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[79]  Wagner Meira,et al.  Learning to rank at query-time using association rules , 2008, SIGIR '08.

[80]  Jianchang Mao Machine Learning in Online Advertising , 2009, ICEIS.

[81]  Fernando Diaz,et al.  Regularizing query-based retrieval scores , 2007, Information Retrieval.

[82]  Hugh E. Williams,et al.  Fast generation of result snippets in web search , 2007, SIGIR.

[83]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[84]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[85]  Hongyuan Zha,et al.  Incorporating query difference for learning retrieval functions in world wide web search , 2006, CIKM '06.

[86]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[87]  John D. Lafferty,et al.  Cranking: Combining Rankings Using Conditional Probability Models on Permutations , 2002, ICML.

[88]  Tao Qin,et al.  Ranking with multiple hyperplanes , 2007, SIGIR.

[89]  Filip Radlinski,et al.  Query chains: learning to rank from implicit feedback , 2005, KDD '05.

[90]  John Guiver,et al.  Learning to rank with SoftRank and Gaussian processes , 2008, SIGIR '08.

[91]  Tao Qin,et al.  LETOR: Benchmark Dataset for Research on Learning to Rank for Information Retrieval , 2007 .

[92]  Patrick Gallinari,et al.  Ranking with ordered weighted pairwise classification , 2009, ICML '09.

[93]  Tie-Yan Liu,et al.  Time-dependent semantic similarity measure of queries using historical click-through data , 2006, WWW '06.

[94]  Stephen E. Robertson,et al.  On rank-based effectiveness measures and optimization , 2007, Information Retrieval.

[95]  Harry Shum,et al.  Query Dependent Ranking Using K-nearest Neighbor * , 2022 .

[96]  Hang Li,et al.  Ranking refinement and its application to information retrieval , 2008, WWW.

[97]  A. Mathur,et al.  Ranking Experts with Discriminative Probabilistic Models , 2009 .

[98]  Bianca Zadrozny,et al.  Learning and evaluating classifiers under sample selection bias , 2004, ICML.

[99]  Massih-Reza Amini,et al.  Learning to Rank for Collaborative Filtering , 2007, ICEIS.

[100]  W. Bruce Croft,et al.  Direct Maximization of Rank-Based Metrics for Information Retrieval , 2005 .

[101]  Jian-Yun Nie,et al.  Learning to Rank Documents for Ad-Hoc Retrieval with Regularized Models , 2007 .

[102]  Pinar Donmez,et al.  On the local optimality of LambdaRank , 2009, SIGIR.

[103]  Jianfeng Gao,et al.  Linear discriminant model for information retrieval , 2005, SIGIR '05.

[104]  Tapas Kanungo,et al.  Machine Learned Sentence Selection Strategies for Query-Biased Summarization , 2008 .

[105]  Jason D. M. Rennie,et al.  Loss Functions for Preference Levels: Regression with Discrete Ordered Labels , 2005 .

[106]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[107]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[108]  Harris Wu,et al.  The effects of fitness functions on genetic programming-based ranking discovery for Web search: Research Articles , 2004 .

[109]  Jian Hu,et al.  Optimizing search engine revenue in sponsored search , 2009, SIGIR.

[110]  Vassilis Plachouras,et al.  Online learning from click data for sponsored search , 2008, WWW.

[111]  Xindong Wu,et al.  Lecture Notes in Machine Learning , 1994, Informatica.

[112]  Garrison W. Cottrell,et al.  Learning to Retrieve Information , 1995 .

[113]  Stephen E. Robertson,et al.  Okapi at TREC-3 , 1994, TREC.

[114]  Xin Li,et al.  Incorporating robustness into web ranking evaluation , 2009, CIKM.

[115]  M. E. Maron,et al.  On Relevance, Probabilistic Indexing and Information Retrieval , 1960, JACM.

[116]  Jaime G. Carbonell,et al.  Fast learning of document ranking functions with the committee perceptron , 2008, WSDM '08.

[117]  Amnon Shashua,et al.  Ranking with Large Margin Principle: Two Approaches , 2002, NIPS.

[118]  Tie-Yan Liu,et al.  Two-Layer Generalization Analysis for Ranking Using Rademacher Average , 2010, NIPS.

[119]  Emine Yilmaz,et al.  Document selection methodologies for efficient and effective learning-to-rank , 2009, SIGIR.

[120]  Fredric C. Gey,et al.  Inferring probability of relevance using the method of logistic regression , 1994, SIGIR '94.

[121]  Dan Roth,et al.  An Unsupervised Learning Algorithm for Rank Aggregation , 2007, ECML.

[122]  Wolfgang Nejdl,et al.  MailRank: using ranking for spam detection , 2005, CIKM '05.

[123]  Chris Buckley,et al.  OHSUMED: an interactive retrieval evaluation and new large test collection for research , 1994, SIGIR '94.

[124]  Christopher J. C. Burges,et al.  High accuracy retrieval with multiple nested ranker , 2006, SIGIR.

[125]  Weiguo Fan,et al.  Discovery of context-specific ranking functions for effective information retrieval using genetic programming , 2004, IEEE Transactions on Knowledge and Data Engineering.

[126]  Filip Radlinski,et al.  Active exploration for learning rankings from clickthrough data , 2007, KDD '07.

[127]  John D. Lafferty,et al.  Model-based feedback in the language modeling approach to information retrieval , 2001, CIKM '01.

[128]  G. Lugosi,et al.  Ranking and empirical minimization of U-statistics , 2006, math/0603123.

[129]  Javed A. Aslam,et al.  A unified model for metasearch and the efficient evaluation of retrieval systems via the hedge algorithm , 2003, SIGIR '03.

[130]  Anonymous Author Robust Reductions from Ranking to Classification , 2006 .

[131]  David C. Gibbon,et al.  Support vector machines: relevance feedback and information retrieval , 2002, Inf. Process. Manag..

[132]  P. Bollmann,et al.  INFORMATION RETRIEVAL BASED ON AXIOMATIC DECISION THEORY , 1991 .

[133]  Hwanjo Yu,et al.  SVM selective sampling for ranking with application to data retrieval , 2005, KDD '05.

[134]  Mehryar Mohri,et al.  An Efficient Reduction of Ranking to Classification , 2007, COLT.

[135]  Shuming Shi,et al.  Title extraction from bodies of HTML documents and its application to web page retrieval , 2005, SIGIR '05.

[136]  Mihai Surdeanu,et al.  Learning to Rank Answers on Large Online QA Collections , 2008, ACL.

[137]  Somnath Banerjee,et al.  Learning to rank for quantity consensus queries , 2009, SIGIR.

[138]  Zhaohui Zheng,et al.  Session Based Click Features for Recency Ranking , 2010, AAAI.

[139]  Glenn Fung,et al.  Learning Rankings via Convex Hull Separation , 2005, NIPS.

[140]  Wei Chu,et al.  New approaches to support vector ordinal regression , 2005, ICML.

[141]  R. Plackett The Analysis of Permutations , 1975 .

[142]  Avinava Dubey,et al.  Efficient and Accurate Local Learning for Ranking , 2009 .

[143]  T. Minka Selection bias in the LETOR datasets , 2008 .

[144]  M. Kay Language Models , 2006 .

[145]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[146]  Yiqun Liu,et al.  Is learning to rank effective for Web search ? , 2009 .

[147]  Hector Garcia-Molina,et al.  Combating Web Spam with TrustRank , 2004, VLDB.

[148]  Stefan Rüping,et al.  Ranking interesting subgroups , 2009, ICML '09.

[149]  Stephen E. Robertson,et al.  Deep versus shallow judgments in learning to rank , 2009, SIGIR.

[150]  Chao Liu,et al.  Efficient multiple-click models in web search , 2009, WSDM '09.

[151]  Tie-Yan Liu,et al.  Adapting ranking SVM to document retrieval , 2006, SIGIR.

[152]  Brendan J. Frey,et al.  Structured ranking learning using cumulative distribution networks , 2008, NIPS.

[153]  In-Ho Kang,et al.  Query type classification for web document retrieval , 2003, SIGIR.

[154]  John D. Lafferty,et al.  A risk minimization framework for information retrieval , 2006, Inf. Process. Manag..

[155]  David Hawking,et al.  Overview of the TREC 2003 Web Track , 2003, TREC.

[156]  Tao Tao,et al.  An exploration of proximity measures in information retrieval , 2007, SIGIR.

[157]  Peter L. Bartlett,et al.  Rademacher and Gaussian Complexities: Risk Bounds and Structural Results , 2003, J. Mach. Learn. Res..

[158]  William W. Cohen,et al.  A Meta-Learning Approach for Robust Rank Learning , 2008 .

[159]  Leonardo Rigutini SortNet: Learning To Rank By a Neural-Based Sorting Algorithm , 2008 .

[160]  Kunle Olukotun,et al.  Map-Reduce for Machine Learning on Multicore , 2006, NIPS.

[161]  William N. Venables,et al.  Modern Applied Statistics with S , 2010 .

[162]  Thomas Hofmann,et al.  Support Vector Machines for Multiple-Instance Learning , 2002, NIPS.

[163]  Chao Liu,et al.  Click chain model in web search , 2009, WWW '09.

[164]  C. Burges,et al.  Learning to Rank Using Classification and Gradient Boosting , 2008 .

[165]  Maksims Volkovs,et al.  BoltzRank: learning to maximize expected ranking gain , 2009, ICML '09.

[166]  Andrew Trotman,et al.  Learning to Rank , 2005, Information Retrieval.

[167]  Tie-Yan Liu,et al.  Directly optimizing evaluation measures in learning to rank , 2008, SIGIR.

[168]  Hongyuan Zha,et al.  A regression framework for learning ranking functions using relative relevance judgments , 2007, SIGIR.

[169]  Azadeh Shakery,et al.  Relevance Propagation for Topic Distillation UIUC TREC 2003 Web Track Experiments , 2003, TREC.

[170]  Eric Brill,et al.  Improving web search ranking by incorporating user behavior information , 2006, SIGIR.

[171]  Olivier Chapelle,et al.  A dynamic bayesian network click model for web search ranking , 2009, WWW '09.

[172]  Stephen E. Robertson,et al.  Optimisation methods for ranking functions with multiple parameters , 2006, CIKM '06.

[173]  Tao Qin,et al.  A study of relevance propagation for web search , 2005, SIGIR '05.

[174]  Sreenivas Gollapudi,et al.  Diversifying search results , 2009, WSDM '09.

[175]  Ingemar J. Cox,et al.  The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical experiments , 2000, IEEE Trans. Image Process..

[176]  Weiguo Fan,et al.  On linear mixture of expert approaches to information retrieval , 2006, Decis. Support Syst..

[177]  Lidan Wang,et al.  Learning to efficiently rank , 2010, SIGIR.

[178]  Andrei Broder,et al.  A taxonomy of web search , 2002, SIGF.

[179]  Nick Craswell,et al.  An experimental comparison of click position-bias models , 2008, WSDM '08.

[180]  Weiguo Fan,et al.  Genetic Programming-Based Discovery of Ranking Functions for Effective Web Search , 2005, J. Manag. Inf. Syst..

[181]  Tie-Yan Liu Are Algorithms Directly Optimizing IR Measures Really Direct , 2008 .

[182]  Weiguo Fan,et al.  A generic ranking function discovery framework by genetic programming for information retrieval , 2004, Inf. Process. Manag..

[183]  Tie-Yan Liu,et al.  Listwise approach to learning to rank: theory and algorithm , 2008, ICML '08.

[184]  Stephen E. Robertson,et al.  SoftRank: optimizing non-smooth rank metrics , 2008, WSDM '08.

[185]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[186]  Sreenivas Gollapudi,et al.  An axiomatic approach for result diversification , 2009, WWW '09.

[187]  Tie-Yan Liu,et al.  Generalization analysis of listwise learning-to-rank algorithms , 2009, ICML '09.

[188]  Brian D. Davison,et al.  Topical link analysis for web search , 2006, SIGIR.

[189]  Iadh Ounis,et al.  A study of parameter tuning for term frequency normalization , 2003, CIKM '03.

[190]  Thorsten Joachims,et al.  A support vector method for multivariate performance measures , 2005, ICML.

[191]  Michael Collins,et al.  Ranking Algorithms for Named Entity Extraction: Boosting and the VotedPerceptron , 2002, ACL.

[192]  Chiranjib Bhattacharyya,et al.  Structured learning for non-smooth ranking losses , 2008, KDD.

[193]  John Dunnion,et al.  ProbFuse: a probabilistic approach to data fusion , 2006, SIGIR.

[194]  Qiang Yang,et al.  Exploiting the hierarchical structure for link analysis , 2005, SIGIR '05.

[195]  Tie-Yan Liu Learning to Rank for Information Retrieval , 2009, Found. Trends Inf. Retr..

[196]  Wei Chu,et al.  Preference learning with Gaussian processes , 2005, ICML.

[197]  Matthew Richardson,et al.  Predicting clicks: estimating the click-through rate for new ads , 2007, WWW '07.

[198]  Berkant Barla Cambazoglu,et al.  Early exit optimizations for additive machine learned ranking systems , 2010, WSDM '10.

[199]  Tao Qin,et al.  How to Make LETOR More Useful and Reliable , 2008 .

[200]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[201]  Colin Campbell,et al.  Bayes Point Machines , 2001, J. Mach. Learn. Res..

[202]  Tie-Yan Liu,et al.  Statistical Consistency of Top-k Ranking , 2009, NIPS.

[203]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[204]  Mingrui Wu,et al.  Gradient descent optimization of smoothed information retrieval metrics , 2010, Information Retrieval.

[205]  Virgil Pavlu,et al.  Large Scale IR Evaluation. , 2008 .

[206]  Gilad Mishne,et al.  Towards recency ranking in web search , 2010, WSDM '10.

[207]  Thorsten Joachims,et al.  Evaluating Retrieval Performance Using Clickthrough Data , 2003, Text Mining.

[208]  Tao Qin,et al.  Query-level loss functions for information retrieval , 2008, Inf. Process. Manag..

[209]  Tao Qin,et al.  FRank: a ranking method with fidelity loss , 2007, SIGIR.

[210]  André Elisseeff,et al.  Stability and Generalization , 2002, J. Mach. Learn. Res..

[211]  Fredric C. Gey,et al.  Probabilistic retrieval based on staged logistic regression , 1992, SIGIR '92.

[212]  Azadeh Shakery,et al.  A probabilistic relevance propagation model for hypertext retrieval , 2006, CIKM '06.

[213]  Tie-Yan Liu,et al.  Learning to Rank for Information Retrieval , 2011 .

[214]  Tao Tao,et al.  Regularized estimation of mixture models for robust pseudo-relevance feedback , 2006, SIGIR.

[215]  James Allan,et al.  Evaluation over thousands of queries , 2008, SIGIR '08.

[216]  J. Marden Analyzing and Modeling Rank Data , 1996 .

[217]  Yi-Hsuan Yang,et al.  ContextSeer: context search and recommendation at query time for shared consumer photos , 2008, ACM Multimedia.

[218]  Steffen Bickel,et al.  Dirichlet-Enhanced Spam Filtering based on Biased Samples , 2006, NIPS.

[219]  Chao Liu,et al.  BBM: bayesian browsing model from petabyte-scale data , 2009, KDD.

[220]  Massih-Reza Amini,et al.  Generalization error bounds for classifiers trained with interdependent data , 2005, NIPS.

[221]  Charles L. A. Clarke,et al.  Novelty and diversity in information retrieval evaluation , 2008, SIGIR '08.

[222]  Zoubin Ghahramani,et al.  Learning from labeled and unlabeled data with label propagation , 2002 .

[223]  Manfred K. Warmuth,et al.  Additive versus exponentiated gradient updates for linear prediction , 1995, STOC '95.

[224]  Robert L. Mercer,et al.  The Mathematics of Statistical Machine Translation: Parameter Estimation , 1993, CL.

[225]  Susan T. Dumais,et al.  Learning user interaction models for predicting web search result preferences , 2006, SIGIR.

[226]  Mehryar Mohri,et al.  Magnitude-preserving ranking algorithms , 2007, ICML '07.

[227]  James Allan,et al.  Minimal test collections for retrieval evaluation , 2006, SIGIR.

[228]  Zheng Chen,et al.  Knowledge transfer for cross domain learning to rank , 2010, Information Retrieval.

[229]  Gábor Lugosi,et al.  Introduction to Statistical Learning Theory , 2004, Advanced Lectures on Machine Learning.

[230]  Wei Yuan,et al.  Smoothing clickthrough data for web search ranking , 2009, SIGIR.

[231]  Jen-Yuan Yeh,et al.  Learning to rank for information retrieval using layered multi-population genetic programming , 2012, 2012 IEEE International Conference on Computational Intelligence and Cybernetics (CyberneticsCom).

[232]  Stephen E. Robertson,et al.  Overview of the Okapi projects , 1997, J. Documentation.

[233]  Thore Graepel,et al.  Large Margin Rank Boundaries for Ordinal Regression , 2000 .

[234]  Min Zhao,et al.  Ranking definitions with supervised learning methods , 2005, WWW '05.

[235]  Cynthia Rudin,et al.  Ranking with a P-Norm Push , 2006, COLT.

[236]  Tie-Yan Liu,et al.  Learning to rank: from pairwise approach to listwise approach , 2007, ICML '07.

[237]  Silviu Guiasu,et al.  The principle of maximum entropy , 1985 .

[238]  Wei-Pang Yang,et al.  Learning to Rank for Information Retrieval Using Genetic Programming , 2007 .

[239]  Shivani Agarwal,et al.  Stability and Generalization of Bipartite Ranking Algorithms , 2005, COLT.

[240]  Hongyuan Zha,et al.  A General Boosting Method and its Application to Learning Ranking Functions for Web Search , 2007, NIPS.

[241]  T. Salakoski,et al.  Learning to Rank with Pairwise Regularized Least-Squares , 2007 .

[242]  Ramesh Nallapati,et al.  Discriminative models for information retrieval , 2004, SIGIR '04.

[243]  Qiang Wu,et al.  McRank: Learning to Rank Using Multiple Classification and Gradient Boosting , 2007, NIPS.

[244]  John D. Lafferty,et al.  Conditional Models on the Ranking Poset , 2002, NIPS.