Multi-task least-squares support vector machines

There are often the underlying cross relatedness amongst multiple tasks, which is discarded directly by traditional single-task learning methods. Since multi-task learning can exploit these relatedness to further improve the performance, it has attracted extensive attention in many domains including multimedia. It has been shown through a meticulous empirical study that the generalization performance of Least-Squares Support Vector Machine (LS-SVM) is comparable to that of SVM. In order to generalize LS-SVM from single-task to multi-task learning, inspired by the regularized multi-task learning (RMTL), this study proposes a novel multi-task learning approach, multi-task LS-SVM (MTLS-SVM). Similar to LS-SVM, one only solves a convex linear system in the training phrase, too. What’s more, we unify the classification and regression problems in an efficient training algorithm, which effectively employs the Krylow methods. Finally, experimental results on school and dermatology validate the effectiveness of the proposed approach.

[1]  M. Pontil,et al.  A Convex Optimization Approach to Modeling Consumer Heterogeneity in Conjoint Estimation , 2007 .

[2]  Patrick Gallinari,et al.  A Learning to Rank framework applied to text-image retrieval , 2011, Multimedia Tools and Applications.

[3]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[4]  Charles A. Micchelli,et al.  Kernels for Multi--task Learning , 2004, NIPS.

[5]  Shuo Xu,et al.  Learn from the Information Contained in the False Splice Sites as well as in the True Splice Sites using SVM , 2007 .

[6]  Charles A. Micchelli,et al.  Universal Multi-Task Kernels , 2008, J. Mach. Learn. Res..

[7]  Ya Zhang,et al.  Multi-task learning for boosting with application to web search ranking , 2010, KDD.

[8]  Jonathan Baxter,et al.  A Model of Inductive Bias Learning , 2000, J. Artif. Intell. Res..

[9]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[10]  Sebastian Thrun,et al.  Learning to Learn , 1998, Springer US.

[11]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[12]  Lin Li,et al.  Multi-output least-squares support vector regression machines , 2013, Pattern Recognit. Lett..

[13]  Tony Jebara,et al.  Multi-task feature and kernel selection for SVMs , 2004, ICML.

[14]  Greg M. Allenby,et al.  A Hierarchical Bayes Model of Primary and Secondary Demand , 1998 .

[15]  Shai Ben-David,et al.  A theoretical framework for learning from a pool of disparate data sources , 2002, KDD.

[16]  Johan A. K. Suykens,et al.  Optimized fixed-size kernel models for large data sets , 2010, Comput. Stat. Data Anal..

[17]  Gunnar Rätsch,et al.  A Mathematical Programming Approach to the Kernel Fisher Algorithm , 2000, NIPS.

[18]  Gavin C. Cawley,et al.  Fast exact leave-one-out cross-validation of sparse least-squares support vector machines , 2004, Neural Networks.

[19]  R. Shah,et al.  Least Squares Support Vector Machines , 2022 .

[20]  Alexander Gammerman,et al.  Ridge Regression Learning Algorithm in Dual Variables , 1998, ICML.

[21]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[22]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[23]  Johan A. K. Suykens,et al.  Least squares support vector machine classifiers: a large scale algorithm , 1999 .

[24]  S. Keerthi,et al.  SMO Algorithm for Least-Squares SVM Formulations , 2003, Neural Computation.

[25]  Antonio Torralba,et al.  Sharing features: efficient boosting procedures for multiclass object detection , 2004, CVPR 2004.

[26]  Federico Girosi,et al.  An Equivalence Between Sparse Approximation and Support Vector Machines , 1998, Neural Computation.

[27]  Gavin C. Cawley,et al.  Leave-One-Out Cross-Validation Based Model Selection Criteria for Weighted LS-SVMs , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[28]  Charles A. Micchelli,et al.  Learning Multiple Tasks with Kernel Methods , 2005, J. Mach. Learn. Res..

[29]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[30]  Johan A. K. Suykens,et al.  Bayesian Framework for Least-Squares Support Vector Machine Classifiers, Gaussian Processes, and Kernel Fisher Discriminant Analysis , 2002, Neural Computation.

[31]  Peter E. Rossi,et al.  Marketing models of consumer heterogeneity , 1998 .

[32]  Tom Heskes,et al.  Task Clustering and Gating for Bayesian Multitask Learning , 2003, J. Mach. Learn. Res..

[33]  Xiaolong Wang,et al.  Novel knowledge-based mean force potential at the profile level , 2006, BMC Bioinformatics.

[34]  K. Johana,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2022 .

[35]  Matthias W. Seeger,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[36]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[37]  S. Sathiya Keerthi,et al.  Semi-supervised multi-task learning of structured prediction models for web information extraction , 2011, CIKM '11.

[38]  Hsuan-Tien Lin.  A Study on Sigmoid Kernels for SVM and the Training of non-PSD Kernels by SMO-type Methods , 2005 .

[39]  Jia-Lien Hsu,et al.  A cross-modal method of labeling music tags , 2011, Multimedia Tools and Applications.

[40]  Chih-Jen Lin,et al.  Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel , 2003, Neural Computation.

[41]  Johan A. K. Suykens,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2004, Machine Learning.

[42]  Rich Caruana,et al.  Multitask Learning , 1997, Machine Learning.

[43]  W. Press,et al.  Numerical Recipes in C++: The Art of Scientific Computing (2nd edn)1 Numerical Recipes Example Book (C++) (2nd edn)2 Numerical Recipes Multi-Language Code CD ROM with LINUX or UNIX Single-Screen License Revised Version3 , 2003 .

[44]  Bernhard Schölkopf,et al.  The connection between regularization operators and support vector kernels , 1998, Neural Networks.

[45]  T. Ben-David,et al.  Exploiting Task Relatedness for Multiple , 2003 .

[46]  Tao Xiong,et al.  A combined SVM and LDA approach for classification , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[47]  Massimiliano Pontil,et al.  Convex multi-task feature learning , 2008, Machine Learning.

[48]  Tong Zhang,et al.  A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , 2005, J. Mach. Learn. Res..

[49]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[50]  Massimiliano Pontil,et al.  Regularized multi--task learning , 2004, KDD.

[51]  Tom Heskes,et al.  Empirical Bayes for Learning to Learn , 2000, ICML.

[52]  Xin An,et al.  [Multiple dependent variables LS-SVM regression algorithm and its application in NIR spectral quantitative analysis]. , 2009, Guang pu xue yu guang pu fen xi = Guang pu.

[53]  Jonathan Baxter,et al.  A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling , 1997, Machine Learning.

[54]  Jieping Ye,et al.  SVM versus Least Squares SVM , 2007, AISTATS.

[55]  Bradley P. Coe,et al.  SIGMA: A System for Integrative Genomic Microarray Analysis of Cancer Genomes , 2006, BMC Genomics.

[56]  Xin An,et al.  [Multi-task least-squares support vector regression machines and their applications in NIR spectral analysis]. , 2011, Guang pu xue yu guang pu fen xi = Guang pu.

[57]  Lin Li,et al.  Semi-supervised Least-squares Support Vector Regression Machines ★ , 2011 .