Closed-form projection operator wavelet kernels in support vector learning for nonlinear dynamical systems identification

As a special idempotent operator, the projection operator plays a crucial role in the Spectral Decomposition Theorem for linear operators in Hilbert space. In this paper, an innovative orthogonal projection operator wavelet kernel is developed for support vector learning. In the framework of multi-resolution analysis, the proposed wavelet kernel can easily fulfill the multi-scale, multidimensional learning to estimate complex dependencies. The peculiar advantage of the wavelet kernel developed in this paper lies in its expressivity in closed-form, which greatly facilitates its application in kernel learning. To our best knowledge, it is the first closed-form orthogonal projection wavelet kernel in the literature. In the scenario of linear programming support vector learning, the proposed closed-form projection operator wavelet kernel is used to identify a parallel model of a benchmark nonlinear dynamical system. A simulation study confirms its superiority in model accuracy and sparsity.

[1]  B. Schölkopf,et al.  Linear programs for automatic accuracy control in regression. , 1999 .

[2]  Stéphane Mallat,et al.  A Wavelet Tour of Signal Processing - The Sparse Way, 3rd Edition , 2008 .

[3]  Zhao Lu,et al.  Linear programming support vector regression with wavelet kernel: A new approach to nonlinear dynamical systems identification , 2009, Math. Comput. Simul..

[4]  王斌,et al.  基于鲁棒小波 ν -支持向量机的产品销售预测模型 , 2009 .

[5]  Zhao Lu,et al.  Linear Programming SVM-ARMA $_{\rm 2K}$ With Application in Engine System Identification , 2011, IEEE Transactions on Automation Science and Engineering.

[6]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[7]  A. Atiya,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2005, IEEE Transactions on Neural Networks.

[8]  Bo-Suk Yang,et al.  Wavelet support vector machine for induction machine fault diagnosis based on transient current signal , 2008, Expert Syst. Appl..

[9]  Y. Meyer,et al.  Wavelets: Calderón-Zygmund and Multilinear Operators , 1997 .

[10]  Nouna Kettaneh,et al.  Statistical Modeling by Wavelets , 1999, Technometrics.

[11]  O. Nelles.  Nonlinear System Identification , 2001 .

[12]  WuQi.  The forecasting model based on wavelet ν-support vector machine , 2009 .

[13]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[14]  John L. Junkins,et al.  Multi-Resolution Methods for Modeling and Control of Dynamical Systems , 2008 .

[15]  Gilbert G. Walter,et al.  Wavelets in Closed Forms , 2001 .

[16]  Ljubo B. Vlacic,et al.  Learning and Soft Computing, Support Vector Machines, Neural Networks, and Fuzzy Logic Models, Vojislav Kecman; MIT Press, Cambridge, MA, 2001, ISBN 0-262-11255-8, 2001, pp 578 , 2002, Neurocomputing.

[17]  Sujit K. Ghosh,et al.  Essential Wavelets for Statistical Applications and Data Analysis , 2001, Technometrics.

[18]  Vojislav Kecman,et al.  Support vectors selection by linear programming , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[19]  S. Billings,et al.  Long term prediction of non-linear time series using multiresolution wavelet models , 2006 .

[20]  Sterling K. Berberian,et al.  Introduction to Hilbert Space , 1977 .

[21]  Jun Zhang,et al.  Orthonormal wavelets with simple closed-form expressions , 1998, IEEE Trans. Signal Process..

[22]  Li Zhang,et al.  Wavelet support vector machine , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  Jochen Krebs,et al.  Support vector regression for the solution of linear integral equations , 2011 .

[24]  Marti A. Hearst.  Trends & Controversies: Support Vector Machines , 1998, IEEE Intell. Syst..

[25]  Anna T. Lawniczak,et al.  Features extraction via wavelet kernel PCA for data classification , 2010, 2010 IEEE International Workshop on Machine Learning for Signal Processing.

[26]  J. Weston,et al.  Support vector regression with ANOVA decomposition kernels , 1999 .

[27]  Gianluca Bontempi,et al.  Conditionally dependent strategies for multiple-step-ahead prediction in local learning , 2011 .

[28]  Qi Wu,et al.  The forecasting model based on wavelet nu-support vector machine , 2009, Expert Syst. Appl..

[29]  G. Walter,et al.  Wavelets and Other Orthogonal Systems , 2018 .

[30]  Roland Opfer,et al.  Multiscale kernels , 2006, Adv. Comput. Math..

[31]  Hava T. Siegelmann,et al.  Computational capabilities of recurrent NARX neural networks , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[32]  Hong Yan,et al.  Framelet Kernels With Applications to Support Vector Regression and Regularization Networks , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[33]  José Luis Rojo-Álvarez,et al.  Support Vector Machines for Nonlinear Kernel ARMA System Identification , 2006, IEEE Transactions on Neural Networks.

[34]  Sridhar Krishnan,et al.  Wavelet Kernel Principal Component Analysis in Noisy Multiscale Data Classification , 2012 .