Training primal twin support vector regression via unconstrained convex minimization

In this paper, we propose a new unconstrained twin support vector regression model in the primal space (UPTSVR). With the addition of a regularization term in the formulation of the problem, the structural risk is minimized. The proposed formulation solves two smaller sized unconstrained minimization problems having continues, piece-wise quadratic objective functions by gradient based iterative methods. However, since their objective functions contain the non-smooth ‘plus’ function, two approaches are taken: (i) replace the non-smooth ‘plus’ function with their smooth approximate functions; (ii) apply a generalized derivative of the non-smooth ‘plus’ function. They lead to five algorithms whose pseudo-codes are also given. Experimental results obtained on a number of interesting synthetic and real-world benchmark datasets using these algorithms in comparison with the standard support vector regression (SVR) and twin SVR (TSVR) clearly demonstrates the effectiveness of the proposed method.

[1]  Laura Schweitzer,et al.  Advances In Kernel Methods Support Vector Learning , 2016 .

[2]  Madan Gopal,et al.  Application of smoothing technique on twin support vector machines , 2008, Pattern Recognit. Lett..

[3]  F. Girosi,et al.  Nonlinear prediction of chaotic time series using support vector machines , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[4]  R. Eubank.  Nonparametric Regression and Spline Smoothing , 1999 .

[5]  Olivier Chapelle,et al.  Training a Support Vector Machine in the Primal , 2007, Neural Computation.

[6]  Ping Zhong,et al.  Training twin support vector regression via linear programming , 2012, Neural Computing and Applications.

[7]  Xiaowei Yang,et al.  A Kernel Fuzzy c-Means Clustering-Based Fuzzy Support Vector Machine Algorithm for Classification Problems With Outliers or Noises , 2011, IEEE Transactions on Fuzzy Systems.

[8]  Lutgarde M. C. Buydens,et al.  Using support vector machines for time series prediction , 2003 .

[9]  Jian Yang,et al.  Smooth twin support vector regression , 2010, Neural Computing and Applications.

[10]  Xinjun Peng,et al.  Primal twin support vector regression and its sparse approximation , 2010, Neurocomputing.

[11]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1971 .

[12]  Xinjun Peng,et al.  Efficient twin parametric insensitive support vector regression model , 2012, Neurocomputing.

[13]  Guilherme A. Barreto,et al.  NONLINEAR SYSTEM IDENTIFICATION USING LOCAL ARX MODELS BASED ON THE SELF-ORGANIZING MAP , 2008 .

[14]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[15]  Olvi L. Mangasarian,et al.  Multisurface proximal support vector machine classification via generalized eigenvalues , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  J. Hiriart-Urruty,et al.  Generalized Hessian matrix and second-order optimality conditions for problems withC1,1 data , 1984 .

[17]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Jing Zhao,et al.  Twin least squares support vector regression , 2013, Neurocomputing.

[19]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[20]  Yuan-Hai Shao,et al.  An ε-twin support vector machine for regression , 2012, Neural Computing and Applications.

[21]  Reshma Khemchandani,et al.  Twin Support Vector Machines for Pattern Classification , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[23]  D. Haussler,et al.  Knowledge-based analysis of microarray gene expression , 2000 .

[24]  S. Balasundaram,et al.  Smooth Newton method for implicit Lagrangian twin support vector regression , 2013, Int. J. Knowl. Based Intell. Eng. Syst..

[25]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[26]  Olvi L. Mangasarian,et al.  A finite newton method for classification , 2002, Optim. Methods Softw..

[27]  Xinjun Peng,et al.  TSVR: An efficient Twin Support Vector Machine for regression , 2010, Neural Networks.

[28]  Yitian Xu,et al.  A weighted twin support vector regression , 2012, Knowl. Based Syst..

[29]  George E. P. Box,et al.  Time Series Analysis: Forecasting and Control , 1977 .

[30]  S. García,et al.  An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons , 2008 .

[31]  S. Balasundaram,et al.  On implicit Lagrangian twin support vector regression by Newton method , 2014, Int. J. Comput. Intell. Syst..

[32]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[33]  Yuh-Jye Lee,et al.  2-SSVR : A Smooth Support Vector Machine for 2-insensitive Regression , 2004 .

[34]  Glenn Fung,et al.  Finite Newton method for Lagrangian support vector machine classification , 2003, Neurocomputing.

[35]  A. Gretton,et al.  Support vector regression for black-box system identification , 2001, Proceedings of the 11th IEEE Signal Processing Workshop on Statistical Signal Processing (Cat. No.01TH8563).

[36]  Yuh-Jye Lee,et al.  SSVM: A Smooth Support Vector Machine for Classification , 2001, Comput. Optim. Appl..

[37]  C. M. Bishop,et al.  Improvements on Twin Support Vector Machines , 2011 .

[38]  S. Balasundaram,et al.  Training Lagrangian twin support vector regression via unconstrained convex minimization , 2014, Knowl. Based Syst..

[39]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[40]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[41]  Sheng Chen,et al.  Sparse support vector regression based on orthogonal forward selection for the generalised kernel model , 2006, Neurocomputing.

[42]  David R. Musicant,et al.  Lagrangian Support Vector Machines , 2001, J. Mach. Learn. Res..

[43]  S. Balasundaram,et al.  On Lagrangian twin support vector regression , 2012, Neural Computing and Applications.

[44]  Yuh-Jye Lee,et al.  epsilon-SSVR: A Smooth Support Vector Machine for epsilon-Insensitive Regression , 2005, IEEE Trans. Knowl. Data Eng..

[45]  Madan Gopal,et al.  Least squares twin support vector machines for pattern classification , 2009, Expert Syst. Appl..

[46]  Johan A. K. Suykens,et al.  Bankruptcy prediction with least squares support vector machine classifiers , 2003, 2003 IEEE International Conference on Computational Intelligence for Financial Engineering, 2003. Proceedings..