Support Vector Machines: A Nonlinear Modelling and Control Perspective

In recent years neural networks as multilayer perceptrons and radial basis function networks have been frequently used in a wide range of fields, including control theory, signal processing and nonlinear modelling. A promising new methodology is Support Vector Machines (SVM), which has been originally introduced by Vapnik within the area of statistical learning theory and structural risk minimization. SVM approaches to classification, nonlinear function and density estimation lead to convex optimization problems, typically quadratic programming. However, due to their non-parametric nature, the present SVM methods were basically restricted to static problems. We discuss a method of least squares support vector machines (LS-SVM), which has been extended to recurrent models and use in optimal control problems. We explain how robust nonlinear estimation and sparse approximation can be done by means of this kernel based technique. A short overview of hyperparameter tuning methods is given. SVM methods are able to learn and generalize well in large dimensional input spaces and have outperformed many existing methods on benchmark data sets. Its full potential in a dynamical systems and control context remains to be explored.

[1]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[2]  Sandro Ridella,et al.  Circular backpropagation networks for classification , 1997, IEEE Trans. Neural Networks.

[3]  Johan A. K. Suykens,et al.  NLq Theory: A Neural Control Framework with Global Asymptotic Stability Criteria , 1997, Neural Networks.

[4]  Mathukumalli Vidyasagar,et al.  A Theory of Learning and Generalization , 1997 .

[5]  Robert M. Sanner,et al.  Gaussian Networks for Direct Adaptive Control , 1991, 1991 American Control Conference.

[6]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[7]  V. Vapnik.  The Support Vector Method of Function Estimation , 1998 .

[8]  Johan A. K. Suykens,et al.  Least squares support vector machine classifiers: a large scale algorithm , 1999 .

[9]  Kumpati S. Narendra,et al.  Gradient methods for the optimization of dynamical systems containing neural networks , 1991, IEEE Trans. Neural Networks.

[10]  Johan A. K. Suykens,et al.  Weighted least squares support vector machines: robustness and sparse approximation , 2002, Neurocomputing.

[11]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[12]  J. Suykens,et al.  Recurrent least squares support vector machines , 2000 .

[13]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[14]  Carl E. Rasmussen,et al.  In Advances in Neural Information Processing Systems , 2011 .

[15]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[16]  Arthur E. Bryson,et al.  Applied Optimal Control , 1969 .

[17]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[18]  Bernhard Schölkopf,et al.  On a Kernel-Based Method for Pattern Recognition, Regression, Approximation, and Operator Inversion , 1998, Algorithmica.

[19]  Bernhard Schölkopf,et al.  The connection between regularization operators and support vector kernels , 1998, Neural Networks.

[20]  Peter J. Rousseeuw,et al.  Robust regression and outlier detection , 1987 .

[21]  Johan A. K. Suykens,et al.  Training multilayer perceptron classifiers based on a modified support vector method , 1999, IEEE Trans. Neural Networks.

[22]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[23]  Nello Cristianini,et al.  An introduction to Support Vector Machines , 2000 .

[24]  Dragoslav D. Šiljak,et al.  Robust stability of discrete systems , 1988 .

[25]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[26]  Alexander Gammerman,et al.  Ridge Regression Learning Algorithm in Dual Variables , 1998, ICML.

[27]  Federico Girosi,et al.  An Equivalence Between Sparse Approximation and Support Vector Machines , 1998, Neural Computation.

[28]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[29]  Gene H. Golub,et al.  Matrix computations , 1983 .

[30]  C. A. Desoer,et al.  Nonlinear Systems Analysis , 1978 .

[31]  Johan A. K. Suykens,et al.  Artificial neural networks for modelling and control of non-linear systems , 1995 .

[33]  Vladimir Cherkassky,et al.  Learning from Data: Concepts, Theory, and Methods , 1998 .

[34]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[35]  J. Suykens,et al.  Automatic relevance determination for least squares support vector machine regression , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[36]  Johan A. K. Suykens,et al.  Optimal control by least squares support vector machines , 2001, Neural Networks.

[37]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[38]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[39]  Johan A. K. Suykens,et al.  Static and dynamic stabilizing neural controllers, applicable to transition between equilibrium points , 1994, Neural Networks.

[40]  Johan A. K. Suykens,et al.  Financial time series prediction using least squares support vector machines within the evidence framework , 2001, IEEE Trans. Neural Networks.

[41]  Johan A. K. Suykens,et al.  Sparse approximation using least squares support vector machines , 2000, 2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21st Century. Proceedings (IEEE Cat No.00CH36353).

[42]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[43]  Bernhard Schölkopf,et al.  Comparing support vector machines with Gaussian kernels to radial basis function classifiers , 1997, IEEE Trans. Signal Process..

[44]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[45]  Johan A. K. Suykens,et al.  Nonlinear modeling : advanced black-box techniques , 1998 .

[46]  Thomas Parisini,et al.  Neural networks for feedback feedforward nonlinear control systems , 1994, IEEE Trans. Neural Networks.

[47]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.