Training Lagrangian twin support vector regression via unconstrained convex minimization

In this paper, a new unconstrained convex minimization problem formulation is proposed as the Lagrangian dual of the 2-norm twin support vector regression (TSVR). The proposed formulation leads to two smaller sized unconstrained minimization problems having their objective functions piece-wise quadratic and differentiable. It is further proposed to apply gradient based iterative method for solving them. However, since their objective functions contain the non-smooth 'plus' function, two approaches are taken: (i) either considering their generalized Hessian or introducing a smooth function in place of the 'plus' function, and applying Newton-Armijo algorithm; (ii) obtaining their critical points by functional iterative algorithm. Computational results obtained on a number of synthetic and real-world benchmark datasets clearly illustrate the superiority of the proposed unconstrained Lagrangian twin support vector regression formulation as comparable generalization performance is achieved with much faster learning speed in accordance with the classical support vector regression and TSVR.

[1]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[2]  David R. Musicant,et al.  Lagrangian Support Vector Machines , 2001, J. Mach. Learn. Res..

[3]  Xinjun Peng,et al.  TPMSVM: A novel twin parametric-margin support vector machine for pattern recognition , 2011, Pattern Recognit..

[4]  J. Hiriart-Urruty,et al.  Generalized Hessian matrix and second-order optimality conditions for problems withC1,1 data , 1984 .

[5]  S. Balasundaram,et al.  On Lagrangian twin support vector regression , 2012, Neural Computing and Applications.

[6]  Madan Gopal,et al.  Application of smoothing technique on twin support vector machines , 2008, Pattern Recognit. Lett..

[7]  Yuan-Hai Shao,et al.  An ε-twin support vector machine for regression , 2012, Neural Computing and Applications.

[8]  D Haussler,et al.  Knowledge-based analysis of microarray gene expression data by using support vector machines. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[9]  Johan A. K. Suykens,et al.  Bankruptcy prediction with least squares support vector machine classifiers , 2003, 2003 IEEE International Conference on Computational Intelligence for Financial Engineering, 2003. Proceedings..

[10]  Reshma Khemchandani,et al.  Twin Support Vector Machines for Pattern Classification , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Sheng Chen,et al.  Sparse support vector regression based on orthogonal forward selection for the generalised kernel model , 2006, Neurocomputing.

[12]  Yuan-Hai Shao,et al.  Improvements on Twin Support Vector Machines , 2011, IEEE Transactions on Neural Networks.

[13]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  Yuan-Hai Shao,et al.  Least squares twin parametric-margin support vector machine for classification , 2013, Applied Intelligence.

[15]  Xiaowei Yang,et al.  A Kernel Fuzzy c-Means Clustering-Based Fuzzy Support Vector Machine Algorithm for Classification Problems With Outliers or Noises , 2011, IEEE Transactions on Fuzzy Systems.

[16]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[17]  Guilherme A. Barreto,et al.  Nonlinear System Identification Using Local ARX Models Based On The Self-Organazing Map , 2006 .

[18]  Jing Zhao,et al.  Twin least squares support vector regression , 2013, Neurocomputing.

[19]  Hongwei Liu,et al.  Semismooth Newton support vector machine , 2007, Pattern Recognit. Lett..

[20]  Ping Zhong,et al.  Training twin support vector regression via linear programming , 2012, Neural Computing and Applications.

[21]  Olvi L. Mangasarian,et al.  A finite newton method for classification , 2002, Optim. Methods Softw..

[22]  R. Rockafellar.  Conjugate Duality and Optimization , 1987 .

[23]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[24]  Yuh-Jye Lee,et al.  SSVM: A Smooth Support Vector Machine for Classification , 2001, Comput. Optim. Appl..

[25]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[26]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[27]  Glenn Fung,et al.  Finite Newton method for Lagrangian support vector machine classification , 2003, Neurocomputing.

[28]  A. Gretton,et al.  Support vector regression for black-box system identification , 2001, Proceedings of the 11th IEEE Signal Processing Workshop on Statistical Signal Processing (Cat. No.01TH8563).

[29]  Olvi L. Mangasarian,et al.  Multisurface proximal support vector machine classification via generalized eigenvalues , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Madan Gopal,et al.  Least squares twin support vector machines for pattern classification , 2009, Expert Syst. Appl..

[31]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[32]  Xinjun Peng,et al.  TSVR: An efficient Twin Support Vector Machine for regression , 2010, Neural Networks.

[33]  Michael C. Ferris,et al.  Semismooth support vector machines , 2004, Math. Program..

[34]  George E. P. Box,et al.  Time Series Analysis: Forecasting and Control , 1977 .

[35]  Jian Yang,et al.  Smooth twin support vector regression , 2010, Neural Computing and Applications.