Weighted solution path algorithm of support vector regression based on heuristic weight-setting optimization

In the conventional solution path algorithm of support vector regression, the @e-insensitive error of every training sample is equally penalized, which means every sample affects the generalization ability equally. However, in some cases, e.g. time series prediction or noisy function regression, the @e-insensitive error of the sample which could provide more important information should be penalized more heavily. Therefore, the weighted solution path algorithm of support vector regression is proposed in this paper. Error penalty parameter of each training sample is weighted differently, and the whole solution path is modified correspondingly. More importantly, by choosing Arc Tangent function as the prototype to generate weights with various characteristics, a heuristic weight-setting optimization algorithm is proposed to compute the optimal weights using particle swarm optimization (PSO). This method is applicable to different applications. Experiments on time series prediction and noisy function regression are conducted, demonstrating comparable results of the proposed weighted solution path algorithm and encouraging performance of the heuristic weight-setting optimization.

[1]  Gang Wang,et al.  A New Solution Path Algorithm in Support Vector Regression , 2008, IEEE Transactions on Neural Networks.

[2]  Christian W. Dawson,et al.  The effect of different basis functions on a radial basis function network for time series prediction: A comparative study , 2006, Neurocomputing.

[3]  Sheng-De Wang,et al.  Fuzzy support vector machines , 2002, IEEE Trans. Neural Networks.

[4]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[5]  Bernhard Schölkopf,et al.  New Support Vector Algorithms , 2000, Neural Computation.

[6]  Robert Tibshirani,et al.  1-norm Support Vector Machines , 2003, NIPS.

[7]  Yue Shi,et al.  A modified particle swarm optimizer , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[8]  Christian Igel,et al.  Evolutionary tuning of multiple SVM parameters , 2005, ESANN.

[9]  Alexander J. Smola,et al.  Support Vector Method for Function Approximation, Regression Estimation and Signal Processing , 1996, NIPS.

[10]  Maurice Clerc,et al.  The particle swarm - explosion, stability, and convergence in a multidimensional complex space , 2002, IEEE Trans. Evol. Comput..

[11]  Yunqian Ma,et al.  Practical selection of SVM parameters and noise estimation for SVM regression , 2004, Neural Networks.

[12]  Johan A. K. Suykens,et al.  Weighted least squares support vector machines: robustness and sparse approximation , 2002, Neurocomputing.

[13]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[14]  Ji Zhu,et al.  Efficient Computation and Model Selection for the Support Vector Regression , 2007, Neural Computation.

[15]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[16]  S. Rosset,et al.  Piecewise linear regularized solution paths , 2007, 0708.2197.

[17]  James T. Kwok.  Linear Dependency between epsilon and the Input Noise in epsilon-Support Vector Regression , 2001, ICANN.

[18]  Robert Tibshirani,et al.  The Entire Regularization Path for the Support Vector Machine , 2004, J. Mach. Learn. Res..

[19]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[20]  Ulrich Parlitz,et al.  Mixed State Analysis of multivariate Time Series , 2001, Int. J. Bifurc. Chaos.

[21]  Zhenghua Dai,et al.  Noise robust estimates of the largest Lyapunov exponent , 2005 .

[22]  George E. P. Box,et al.  Time Series Analysis: Forecasting and Control , 1977 .

[23]  Gunnar Rätsch,et al.  Predicting Time Series with Support Vector Machines , 1997, ICANN.

[24]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[25]  Hugo Jair Escalante,et al.  Particle Swarm Model Selection , 2009, J. Mach. Learn. Res..

[26]  S. Sathiya Keerthi,et al.  Efficient tuning of SVM hyperparameters using radius/margin bound and iterative algorithms , 2002, IEEE Trans. Neural Networks.

[27]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[28]  Gang Wang,et al.  Two-dimensional solution path for support vector regression , 2006, ICML.

[29]  Zhang Jiang.  Reweighted Robust Support Vector Regression Method , 2005 .

[30]  Stéphane Canu,et al.  Regularization Paths for nu -SVM and nu -SVR , 2007, ISNN.

[31]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[32]  G. Wahba.  Smoothing noisy data with spline functions , 1975 .

[33]  Francis Eng Hock Tay,et al.  Modified support vector machines in financial time series forecasting , 2002, Neurocomputing.

[34]  Xiaowei Yang,et al.  A heuristic weight-setting strategy and iteratively updating algorithm for weighted least-squares support vector regression , 2008, Neurocomputing.

[35]  F. Girosi,et al.  Nonlinear prediction of chaotic time series using support vector machines , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[36]  Andrew W. Smyth,et al.  Neural Network Initialization with Prototypes - Function Approximation in Engineering Mechanics Applications , 2007, 2007 International Joint Conference on Neural Networks.

[37]  B. Schölkopf,et al.  Asymptotically Optimal Choice of ε-Loss for Support Vector Machines , 1998 .

[38]  Hong Gu,et al.  Fuzzy prediction of chaotic time series based on singular value decomposition , 2007, Appl. Math. Comput..

[39]  Q. Henry Wu,et al.  Local prediction of non-linear time series using support vector regression , 2008, Pattern Recognit..

[40]  Hugo Jair Escalante,et al.  Particle Swarm Full Model Selection , 2008 .

[41]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[42]  X. C. Guo,et al.  A novel LS-SVMs hyper-parameter selection based on particle swarm optimization , 2008, Neurocomputing.