A new approach for training Lagrangian support vector regression

In this paper, a novel root finding problem for the Lagrangian support vector regression in 2-norm (LSVR) is formulated in which the number of unknowns becomes the number of training examples. Further, it is proposed to solve it by functional iterative and Newton methods. Under sufficient conditions, we proved their linear rate of convergence. Experiments are performed on a number of synthetic and real-world benchmark datasets, and their results are compared with support vector regression (SVR) and its variants such as least squares SVR and LSVR. Similar generalization performance with improved or comparable learning speed to SVR and its variants demonstrates the usefulness of the proposed formulation solved by the iterative methods.

[1]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[2]  Shai Shalev-Shwartz,et al.  Stochastic dual coordinate ascent methods for regularized loss , 2012, J. Mach. Learn. Res..

[3]  David R. Musicant,et al.  Active set support vector regression , 2004, IEEE Transactions on Neural Networks.

[4]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[5]  Yuh-Jye Lee,et al.  epsilon-SSVR: A Smooth Support Vector Machine for epsilon-Insensitive Regression , 2005, IEEE Trans. Knowl. Data Eng..

[6]  David R. Musicant,et al.  Active Support Vector Machine Classification , 2000, NIPS.

[7]  Guilherme A. Barreto,et al.  NONLINEAR SYSTEM IDENTIFICATION USING LOCAL ARX MODELS BASED ON THE SELF-ORGANIZING MAP , 2008 .

[8]  Gene H. Golub,et al.  Matrix computations , 1983 .

[9]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[10]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[11]  Yuh-Jye Lee,et al.  SSVM: A Smooth Support Vector Machine for Classification , 2001, Comput. Optim. Appl..

[12]  S. Balasundaram,et al.  On Lagrangian support vector regression , 2010, Expert Syst. Appl..

[13]  Xinjun Peng,et al.  Primal twin support vector regression and its sparse approximation , 2010, Neurocomputing.

[14]  Min Wang,et al.  Seeking multi-thresholds directly from support vectors for image segmentation , 2005, Neurocomputing.

[15]  Glenn Fung,et al.  Finite Newton method for Lagrangian support vector machine classification , 2003, Neurocomputing.

[16]  A. Gretton,et al.  Support vector regression for black-box system identification , 2001, Proceedings of the 11th IEEE Signal Processing Workshop on Statistical Signal Processing (Cat. No.01TH8563).

[17]  R. Rockafellar.  Conjugate Duality and Optimization , 1987 .

[18]  Xinjun Peng,et al.  TSVR: An efficient Twin Support Vector Machine for regression , 2010, Neural Networks.

[19]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1971 .

[20]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[21]  Bernardete Ribeiro,et al.  Kernelized based functions with Minkovsky's norm for SVM regression , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[22]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[23]  Reshma Khemchandani,et al.  Twin Support Vector Machines for Pattern Classification , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[25]  Chih-Jen Lin,et al.  A dual coordinate descent method for large-scale linear SVM , 2008, ICML '08.

[26]  George E. P. Box,et al.  Time Series Analysis: Forecasting and Control , 1977 .

[27]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[28]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[29]  Panos M. Pardalos,et al.  Robust Data Mining , 2012 .

[30]  Shiliang Sun,et al.  A review of optimization methodologies in support vector machines , 2011, Neurocomputing.

[31]  S. Balasundaram,et al.  Lagrangian support vector regression via unconstrained convex minimization , 2014, Neural Networks.

[32]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[33]  S. Balasundaram,et al.  Finite Newton method for implicit Lagrangian support vector regression , 2011, Int. J. Knowl. Based Intell. Eng. Syst..

[34]  David R. Musicant,et al.  Lagrangian Support Vector Machines , 2001, J. Mach. Learn. Res..

[35]  Panos M. Pardalos,et al.  Data Mining and Mathematical Programming , 2008 .

[36]  Olvi L. Mangasarian,et al.  A generalized Newton method for absolute value equations , 2009, Optim. Lett..