Linear programming support vector regression with wavelet kernel: A new approach to nonlinear dynamical systems identification

Wavelet theory has a profound impact on signal processing as it offers a rigorous mathematical framework to the treatment of multiresolution problems. The combination of soft computing and wavelet theory has led to a number of new techniques. On the other hand, as a new generation of learning algorithms, support vector regression (SVR) was developed by Vapnik et al. recently, in which e-insensitive loss function was defined as a trade-off between the robust loss function of Huber and one that enables sparsity within the SVs. The use of support vector kernel expansion also provides us a potential avenue to represent nonlinear dynamical systems and underpin advanced analysis. However, for the support vector regression with the standard quadratic programming technique, the implementation is computationally expensive and sufficient model sparsity cannot be guaranteed. In this article, from the perspective of model sparsity, the linear programming support vector regression (LP-SVR) with wavelet kernel was proposed, and the connection between LP-SVR with wavelet kernel and wavelet networks was analyzed. In particular, the potential of the LP-SVR for nonlinear dynamical system identification was investigated.

[1]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[2]  Robert F. Harrison,et al.  A new method for sparsity control in support vector classification and regression , 2001, Pattern Recognit..

[3]  Guangyi Chen,et al.  Pattern recognition with SVM and dual-tree complex wavelets , 2007, Image Vis. Comput..

[4]  Dongkai Yang,et al.  Wavelet kernel Support Vector Machines for sparse approximation , 2006 .

[5]  Li Zhang,et al.  Wavelet support vector machine , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[6]  David P Stoten,et al.  UKACC International Conference on CONTROL , 2008 .

[7]  C. J. Franco,et al.  PREDICCIÓN DE SERIES TEMPORALES USANDO MÁQUINAS DE VECTORES DE SOPORTE TIME SERIES PREDICTION USING SUPPORT VECTOR MACHINES , 2010 .

[8]  B. Schölkopf,et al.  Linear programs for automatic accuracy control in regression. , 1999 .

[9]  Nicola Ancona.  Properties of Support Vector Machines for Regression Properties of Support Vector Machines for Regression , 1999 .

[10]  V. Kecman,et al.  Support vector machines trained by linear programming: theory and application in image compression and data classification , 2000, Proceedings of the 5th Seminar on Neural Network Applications in Electrical Engineering. NEUREL 2000 (IEEE Cat. No.00EX287).

[11]  S. Sitharama Iyengar,et al.  Foundations of Wavelet Networks and Applications , 2002 .

[12]  A. Gretton,et al.  Support vector regression for black-box system identification , 2001, Proceedings of the 11th IEEE Signal Processing Workshop on Statistical Signal Processing (Cat. No.01TH8563).

[13]  Ravi Sankar,et al.  Time Series Prediction Using Support Vector Machines: A Survey , 2009, IEEE Computational Intelligence Magazine.

[14]  Vojislav Kecman,et al.  Learning and Soft Computing: Support Vector Machines, Neural Networks, and Fuzzy Logic Models , 2001 .

[15]  Ralf Herbrich,et al.  Learning Kernel Classifiers , 2001 .

[16]  José Luis Rojo-Álvarez,et al.  Support Vector Machines for Nonlinear Kernel ARMA System Identification , 2006, IEEE Transactions on Neural Networks.

[17]  Qinghua Zhang,et al.  Using wavelet network in nonparametric estimation , 1997, IEEE Trans. Neural Networks.

[18]  Gérard Dreyfus,et al.  Training wavelet networks for nonlinear dynamic input-output modeling , 1998, Neurocomputing.

[19]  Chih-Jen Lin,et al.  Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel , 2003, Neural Computation.

[20]  Stephen A. Billings,et al.  Time series prediction using support vector machines, the orthogonal and the regularized orthogonal least-squares algorithms , 2002, Int. J. Syst. Sci..

[21]  Zhao Lu,et al.  Tracking Control of Nonlinear Systems: a Sliding Mode Design via Chaotic Optimization , 2004, Int. J. Bifurc. Chaos.

[22]  Bing Li,et al.  Optimizing Complex Functions by Chaos Search , 1998, Cybern. Syst..

[23]  Sheng Chen,et al.  Local regularization assisted orthogonal least squares regression , 2006, Neurocomputing.

[24]  Alan F. Murray,et al.  Synaptic Rewiring for Topographic Map Formation , 2008, ICANN.

[25]  David Allingham,et al.  Wavelet reconstruction of nonlinear dynamics , 1998 .

[26]  A. V.DavidSánchez,et al.  Advanced support vector machines and kernel methods , 2003, Neurocomputing.

[27]  Jun Zhang,et al.  Wavelet neural networks for function learning , 1995, IEEE Trans. Signal Process..

[28]  Ralf Herbrich,et al.  Learning Kernel Classifiers: Theory and Algorithms , 2001 .

[29]  R. Harrison,et al.  Support Vector Machines for System Identification , 1998 .

[30]  Ljubo B. Vlacic,et al.  Learning and Soft Computing, Support Vector Machines, Neural Networks, and Fuzzy Logic Models, Vojislav Kecman; MIT Press, Cambridge, MA, 2001, ISBN 0-262-11255-8, 2001, pp 578 , 2002, Neurocomputing.

[31]  Alexander J. Smola,et al.  Learning with Kernels: support vector machines, regularization, optimization, and beyond , 2001, Adaptive computation and machine learning series.

[32]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..