Efficient regularized least-squares algorithms for conditional ranking on relational data

In domains like bioinformatics, information retrieval and social network analysis, one can find learning tasks where the goal consists of inferring a ranking of objects, conditioned on a particular target object. We present a general kernel framework for learning conditional rankings from various types of relational data, where rankings can be conditioned on unseen data objects. We propose efficient algorithms for conditional ranking by optimizing squared regression and ranking loss functions. We show theoretically, that learning with the ranking loss is likely to generalize better than with the regression loss. Further, we prove that symmetry or reciprocity properties of relations can be efficiently enforced in the learned models. Experiments on synthetic and real-world data illustrate that the proposed methods deliver state-of-the-art performance in terms of predictive power and computational efficiency. Moreover, we also show empirically that incorporating symmetry or reciprocity properties can improve the generalization performance.

[1]  Elena Marchiori,et al.  Gaussian interaction profile kernels for predicting drug-target interaction , 2011, Bioinform..

[2]  Alexander Gammerman,et al.  Ridge Regression Learning Algorithm in Dual Variables , 1998, ICML.

[3]  Samy Bengio,et al.  A Discriminative Kernel-Based Approach to Rank Images from Text Queries , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Thomas Hofmann,et al.  Unifying collaborative and content-based filtering , 2004, ICML.

[5]  Richard Simon,et al.  Bias in error estimation when using cross-validation for model selection , 2006, BMC Bioinformatics.

[6]  S. R. Searle,et al.  On Deriving the Inverse of a Sum of Matrices , 1981 .

[7]  Carla D. Moravitz Martin,et al.  Shifted Kronecker Product Systems , 2006, SIAM J. Matrix Anal. Appl..

[8]  Yunming Ye,et al.  MultiRank: co-ranking for objects and relations in multi-relational data , 2011, KDD.

[9]  Tapio Pahikkala,et al.  An efficient algorithm for learning to rank from preference graphs , 2009, Machine Learning.

[10]  Heikki Mannila,et al.  Relational link-based ranking , 2004, VLDB.

[11]  Bernard De Baets,et al.  From learning taxonomies to phylogenetic learning: Integration of 16S rRNA gene data into FAME-based bacterial classification , 2010, BMC Bioinformatics.

[12]  Yoshihiro Yamanishi,et al.  propagation: A fast semisupervised learning algorithm for link prediction , 2009 .

[13]  Yin Yang,et al.  Query by document , 2009, WSDM '09.

[14]  Tapio Salakoski,et al.  Learning monadic and dyadic relations : three case studies in systems biology , 2012 .

[15]  Thomas Hofmann,et al.  Large Margin Methods for Structured and Interdependent Output Variables , 2005, J. Mach. Learn. Res..

[16]  Bernard De Baets,et al.  Cyclic Evaluation of Transitivity of Reciprocal Relations , 2006, Soc. Choice Welf..

[17]  Bernhard Schölkopf,et al.  Joint Kernel Maps , 2005, IWANN.

[18]  Andreas Fischer,et al.  Pairwise support vector machines and their application to large scale problems , 2012, J. Mach. Learn. Res..

[19]  Henk A. van der Vorst,et al.  Bi-CGSTAB: A Fast and Smoothly Converging Variant of Bi-CG for the Solution of Nonsymmetric Linear Systems , 1992, SIAM J. Sci. Comput..

[20]  C. Loan The ubiquitous Kronecker product , 2000 .

[21]  Shivani Agarwal,et al.  Ranking on graph data , 2006, ICML.

[22]  Tommi S. Jaakkola,et al.  Maximum-Margin Matrix Factorization , 2004, NIPS.

[23]  Tapio Salakoski,et al.  A Kernel-Based Framework for Learning Graded Relations From Data , 2011, IEEE Transactions on Fuzzy Systems.

[24]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[25]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[26]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[27]  S. Yau Mathematics and its applications , 2002 .

[28]  Tapio Salakoski,et al.  On Learning and Cross-Validation with Decomposed Nyström Approximation of Kernel Matrix , 2011, Neural Processing Letters.

[29]  Christopher D. Manning,et al.  Using Feature Conjunctions Across Examples for Learning Pairwise Classifiers , 2004, ECML.

[30]  William Stafford Noble,et al.  Kernel methods for predicting protein-protein interactions , 2005, ISMB.

[31]  Eyke Hüllermeier,et al.  Bipartite Ranking through Minimization of Univariate Loss , 2011, ICML.

[32]  Charles Elkan,et al.  Predicting labels for dyadic data , 2010, Data Mining and Knowledge Discovery.

[33]  Tie-Yan Liu,et al.  Listwise approach to learning to rank: theory and algorithm , 2008, ICML '08.

[34]  Tie-Yan Liu,et al.  Learning to rank for information retrieval , 2009, SIGIR.

[35]  Kristian Kersting,et al.  Learning Preferences with Hidden Common Cause Relations , 2009, ECML/PKDD.

[36]  Eyke Hüllermeier,et al.  Label ranking by learning pairwise preferences , 2008, Artif. Intell..

[37]  Phil Culverhouse,et al.  Time to automate identification , 2010, Nature.

[38]  Tao Qin,et al.  Learning to rank relational objects and its application to web search , 2008, WWW.

[39]  S. Sathiya Keerthi,et al.  Efficient algorithms for ranking with SVMs , 2010, Information Retrieval.

[40]  Thorsten Joachims,et al.  Fast Active Exploration for Link-Based Preference Learning Using Gaussian Processes , 2010, ECML/PKDD.

[41]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[42]  Len Fisher,et al.  Rock, paper, scissors : game theory in everyday life , 2008 .

[43]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[44]  Eyke Hüllermeier,et al.  Binary Decomposition Methods for Multipartite Ranking , 2009, ECML/PKDD.

[45]  Alexander J. Smola,et al.  Learning Graph Matching , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  G. Wahba,et al.  Some results on Tchebycheffian spline functions , 1971 .

[47]  Tie-Yan Liu,et al.  Adapting ranking SVM to document retrieval , 2006, SIGIR.

[48]  Tapio Salakoski,et al.  Learning intransitive reciprocal relations with kernel methods , 2010, Eur. J. Oper. Res..

[49]  William I. Gasarch Review of rock, paper, scissors: game theory for everyday life by Len Fisher (Basic Books, 2008) , 2009, SIGA.

[50]  Tapio Salakoski,et al.  Conditional Ranking on Relational Data , 2010, ECML/PKDD.

[51]  Jason Weston,et al.  Protein ranking: from local to global structure in the protein similarity network. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[52]  Filip Radlinski,et al.  A support vector method for optimizing average precision , 2007, SIGIR.

[53]  Yoshihiro Yamanishi,et al.  On Pairwise Kernels: An Efficient Alternative and Generalization Analysis , 2009, PAKDD.

[54]  Tapio Salakoski,et al.  Training linear ranking SVMs in linearithmic time using red-black trees , 2010, Pattern Recognit. Lett..

[55]  Yoshihiro Yamanishi,et al.  Protein network inference from multiple genomic data: a supervised approach , 2004, ISMB/ECCB.

[56]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[57]  Lorenzo Rosasco,et al.  Learning from Examples as an Inverse Problem , 2005, J. Mach. Learn. Res..

[58]  Hisashi Kashima,et al.  Fast and Scalable Algorithms for Semi-supervised Link Prediction on Static and Dynamic Graphs , 2010, ECML/PKDD.

[59]  H. Engl,et al.  Regularization of Inverse Problems , 1996 .

[60]  Michèle Sebag,et al.  Proceedings of the 2010 European conference on Machine learning and knowledge discovery in databases: Part III , 2010 .

[61]  Eugene Galanter,et al.  Handbook of mathematical psychology: I. , 1963 .

[62]  Pierre Geurts,et al.  Inferring biological networks with output kernel trees , 2007, BMC Bioinformatics.

[63]  Thorsten Joachims,et al.  Training linear SVMs in linear time , 2006, KDD '06.

[64]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[65]  Ingo Steinwart,et al.  On the Influence of the Kernel on the Consistency of Support Vector Machines , 2002, J. Mach. Learn. Res..

[66]  Tie-Yan Liu,et al.  Learning to Rank for Information Retrieval , 2011 .

[67]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[68]  William Stafford Noble,et al.  A new pairwise kernel for biological network inference with support vector machines , 2007, BMC Bioinformatics.