Incorporating prior knowledge in support vector regression

Abstract This paper explores the incorporation of prior knowledge in support vector regresion by the addition of constraints. Equality and inequality constraints are studied with the corresponding types of prior knowledge that can be considered for the method. These include particular points with known values, prior knowledge on any derivative of the function either provided by a prior model or available only at some specific points and bounds on the function or any derivative in a given domain. Moreover, a new method for the simultaneous approximation of multiple outputs linked by some prior knowledge is proposed. This method also allows consideration of different types of prior knowledge on single outputs while training on multiple outputs. Synthetic examples show that incorporating a wide variety of prior knowledge becomes easy, as it leads to linear programs, and helps to improve the approximation in difficult cases. The benefits of the method are finally shown on a real-life application, the estimation of in-cylinder residual gas fraction in spark ignition engines, which is representative of numerous situations met in engineering.

[1]  C. Micchelli,et al.  Smoothing and Interpolation in a Convex Subset of a Hilbert Space , 1988 .

[2]  Bernhard Schölkopf,et al.  Incorporating Invariances in Support Vector Learning Machines , 1996, ICANN.

[3]  Simon Haykin,et al.  Support vector machines for dynamic reconstruction of a chaotic system , 1999 .

[4]  Fernando Pérez-Cruz,et al.  SVM multiregression for nonlinear channel estimation in multiple-input multiple-output systems , 2004, IEEE Transactions on Signal Processing.

[5]  J. Weston,et al.  Support vector regression with ANOVA decomposition kernels , 1999 .

[6]  Bernhard Schölkopf,et al.  The connection between regularization operators and support vector kernels , 1998, Neural Networks.

[7]  T. Poggio,et al.  Recognition and Structure from one 2D Model View: Observations on Prototypes, Object Classes and Symmetries , 1992 .

[8]  Robert Andrews,et al.  On the effects of initialising a neural network with prior knowledge , 1999, ICONIP'99. ANZIIS'99 & ANNES'99 & ACNN'99. 6th International Conference on Neural Information Processing. Proceedings (Cat. No.99EX378).

[9]  Francis Eng Hock Tay,et al.  Modified support vector machines in financial time series forecasting , 2002, Neurocomputing.

[10]  Olvi L. Mangasarian,et al.  Nonlinear Knowledge in Kernel Approximation , 2007, IEEE Transactions on Neural Networks.

[11]  J. Weston,et al.  Support vector density estimation , 1999 .

[12]  Gunnar Rätsch,et al.  Predicting Time Series with Support Vector Machines , 1997, ICANN.

[13]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[14]  Petre Stoica,et al.  Decentralized Control , 2018, The Control Systems Handbook.

[15]  Fernando Pérez-Cruz,et al.  Support Vector Regression for the simultaneous learning of a multivariate function and its derivatives , 2005, Neurocomputing.

[16]  Kristin P. Bennett,et al.  Combining support vector and mathematical programming methods for classification , 1999 .

[17]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[18]  Gérard Bloch,et al.  Combining experimental data and physical simulation models in Support Vector learning , 2007 .

[20]  David R. Musicant,et al.  Large Scale Kernel Regression via Linear Programming , 2002, Machine Learning.

[21]  Guillaume Colin,et al.  Residual gas fraction measurement and computation , 2007 .

[22]  Gérard Bloch,et al.  Incorporating prior knowledge in support vector machines for classification: A review , 2008, Neurocomputing.

[23]  Bernhard Schölkopf,et al.  Semiparametric Support Vector and Linear Programming Machines , 1998, NIPS.

[24]  Bernhard Schölkopf,et al.  Kernel Dependency Estimation , 2002, NIPS.

[25]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[26]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[27]  Alexander J. Smola,et al.  Advances in Large Margin Classifiers , 2000 .

[28]  Jude W. Shavlik,et al.  Giving Advice about Preferred Actions to Reinforcement Learners Via Knowledge-Based Kernel Regression , 2005, AAAI.

[29]  Fernando Pérez-Cruz,et al.  Learning a function and its derivative forcing the support vector expansion , 2005, IEEE Signal Processing Letters.

[30]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[31]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[32]  Jude W. Shavlik,et al.  Knowledge-Based Artificial Neural Networks , 1994, Artif. Intell..

[33]  B. Schölkopf,et al.  Linear programs for automatic accuracy control in regression. , 1999 .

[34]  Tor Arne Johansen,et al.  Identification of non-linear systems using empirical data and prior knowledge - an optimization approach , 1996, Autom..

[35]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[36]  Jude W. Shavlik,et al.  Knowledge-Based Kernel Approximation , 2004, J. Mach. Learn. Res..

[37]  Grace Wahba,et al.  Inequality-Constrained Multivariate Smoothing Splines with Application to the Estimation of Posterior Probabilities , 1987 .

[38]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[39]  Hans C. van Houwelingen,et al.  The Elements of Statistical Learning, Data Mining, Inference, and Prediction. Trevor Hastie, Robert Tibshirani and Jerome Friedman, Springer, New York, 2001. No. of pages: xvi+533. ISBN 0‐387‐95284‐5 , 2004 .

[40]  Glenn Fung,et al.  Knowledge-Based Nonlinear Kernel Classifiers , 2003, COLT.

[41]  Thorsten Joachims,et al.  Learning to classify text using support vector machines - methods, theory and algorithms , 2002, The Kluwer international series in engineering and computer science.

[42]  Glenn Fung,et al.  Knowledge-Based Support Vector Machine Classifiers , 2002, NIPS.

[43]  Olvi L. Mangasarian,et al.  Generalized Support Vector Machines , 1998 .

[44]  Rohini K. Srihari,et al.  Incorporating prior knowledge with weighted margin support vector machines , 2004, KDD.