Design of a multiple kernel learning algorithm for LS-SVM by convex programming

As a kernel based method, the performance of least squares support vector machine (LS-SVM) depends on the selection of the kernel as well as the regularization parameter (Duan, Keerthi, & Poo, 2003). Cross-validation is efficient in selecting a single kernel and the regularization parameter; however, it suffers from heavy computational cost and is not flexible to deal with multiple kernels. In this paper, we address the issue of multiple kernel learning for LS-SVM by formulating it as semidefinite programming (SDP). Furthermore, we show that the regularization parameter can be optimized in a unified framework with the kernel, which leads to an automatic process for model selection. Extensive experimental validations are performed and analyzed.

[1]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[2]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[3]  Gunnar Rätsch,et al.  Large Scale Multiple Kernel Learning , 2006, J. Mach. Learn. Res..

[4]  Chuanhou Gao,et al.  Application of Least Squares Support Vector Machines to Predict the Silicon Content in Blast Furnace Hot Metal , 2008 .

[5]  Nello Cristianini,et al.  A statistical framework for genomic data fusion , 2004, Bioinform..

[6]  Johan A. K. Suykens,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2004, Machine Learning.

[7]  Murat Dundar,et al.  A fast iterative algorithm for fisher discriminant using heterogeneous kernels , 2004, ICML.

[8]  Jieping Ye,et al.  Multi-class Discriminant Kernel Learning via Convex Programming , 2008, J. Mach. Learn. Res..

[9]  Alexander J. Smola,et al.  Learning the Kernel with Hyperkernels , 2005, J. Mach. Learn. Res..

[10]  C. Berg,et al.  Harmonic Analysis on Semigroups: Theory of Positive Definite and Related Functions , 1984 .

[11]  Stephen P. Boyd,et al.  Semidefinite Programming , 1996, SIAM Rev..

[12]  Knud D. Andersen,et al.  The Mosek Interior Point Optimizer for Linear Programming: An Implementation of the Homogeneous Algorithm , 2000 .

[13]  Ivor W. Tsang,et al.  Efficient hyperkernel learning using second-order cone programming , 2006, IEEE Trans. Neural Networks.

[14]  Hua Xiang,et al.  Perturbation analysis of generalized saddle point systems , 2006 .

[15]  Johan A. K. Suykens,et al.  L2-norm multiple kernel learning and its application to biomedical data fusion , 2010, BMC Bioinformatics.

[16]  Michael I. Jordan,et al.  Multiple kernel learning, conic duality, and the SMO algorithm , 2004, ICML.

[17]  Dmitrij Frishman,et al.  MIPS: a database for genomes and protein sequences , 1999, Nucleic Acids Res..

[18]  Johan A. K. Suykens,et al.  Low rank updated LS-SVM classifiers for fast variable selection , 2008, Neural Networks.

[19]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .

[20]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[21]  Gavin C. Cawley,et al.  Preventing Over-Fitting during Model Selection via Bayesian Regularisation of the Hyper-Parameters , 2007, J. Mach. Learn. Res..

[22]  S. Sathiya Keerthi,et al.  Evaluation of simple performance measures for tuning SVM hyperparameters , 2003, Neurocomputing.

[23]  Johan A. K. Suykens,et al.  Optimal control by least squares support vector machines , 2001, Neural Networks.

[24]  Andrew McCallum,et al.  Dynamic conditional random fields: factorized probabilistic models for labeling and segmenting sequence data , 2004, J. Mach. Learn. Res..

[25]  Mohamed Cheriet,et al.  Model selection for the LS-SVM. Application to handwriting recognition , 2009, Pattern Recognit..

[26]  Gene H. Golub,et al.  Matrix computations , 1983 .

[27]  Nello Cristianini,et al.  Learning the Kernel Matrix with Semidefinite Programming , 2002, J. Mach. Learn. Res..

[28]  Domenico Conforti,et al.  Kernel based support vector machine via semidefinite programming: Application to medical diagnosis , 2010, Comput. Oper. Res..

[29]  Stephen P. Boyd,et al.  Optimal kernel selection in Kernel Fisher discriminant analysis , 2006, ICML.

[30]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[31]  Xuan Li,et al.  Association of tissue lineage and gene expression: conservatively and differentially expressed genes define common and special functions of tissues , 2010, BMC Bioinformatics.