Motivated Support Vector Regression using Structural Prior Knowledge

It’s known that incorporating prior knowledge into support vector regression (SVR) can help to improve the approximation performance. Most of researches are concerned with the incorporation of knowledge in the form of numerical relationships. Little work, however, has been done to incorporate the prior knowledge on the structural relationships among the variables (referred as to Structural Prior Knowledge, SPK). This paper explores the incorporation of SPK in SVR by constructing appropriate admissible support vector kernel (SV kernel) based on the properties of reproducing kernel (R.K). Three-levels specifications of SPK are studied with the corresponding sub-levels of prior knowledge that can be considered for the method. These include Hierarchical SPK (HSPK), Interactional SPK (ISPK) consisting of independence, global and local interaction, Functional SPK (FSPK) composed of exterior-FSPK and interior-FSPK. A convenient tool for describing the SPK, namely Description Matrix of SPK is introduced. Subsequently, a new SVR, namely Motivated Support Vector Regression (MSVR) whose structure is motivated in part by SPK, is proposed. Synthetic examples show that it is possible to incorporate a wide variety of SPK and helpful to improve the approximation performance in complex cases. The benefits of MSVR are finally shown on a real-life military application, Air-toground battle simulation, which shows great potential for MSVR to the complex military applications. Keywords—admissible support vector kernel, reproducing kernel, structural prior knowledge, motivated support vector regression

[1]  Jun Wang,et al.  A support vector machine with a hybrid kernel and minimal Vapnik-Chervonenkis dimension , 2004, IEEE Transactions on Knowledge and Data Engineering.

[2]  Zhizhong Wang,et al.  Model optimizing and feature selecting for support vector regression in time series forecasting , 2008, Neurocomputing.

[3]  Christopher R. Houck,et al.  A Genetic Algorithm for Function Optimization: A Matlab Implementation , 2001 .

[4]  Grace Wahba,et al.  Spline Models for Observational Data , 1990 .

[5]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[6]  Bernhard Schölkopf,et al.  Prior Knowledge in Support Vector Kernels , 1997, NIPS.

[7]  Guido Smits,et al.  Improved SVM regression using mixtures of kernels , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[8]  Ping Zhu,et al.  Metamodel-based lightweight design of B-pillar with TWB structure via support vector regression , 2010 .

[9]  Junbin Gao,et al.  Mean field method for the support vector machine regression , 2003, Neurocomputing.

[10]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[11]  Bernhard Schölkopf,et al.  Comparing support vector machines with Gaussian kernels to radial basis function classifiers , 1997, IEEE Trans. Signal Process..

[12]  Fernando Pérez-Cruz,et al.  Learning a function and its derivative forcing the support vector expansion , 2005, IEEE Signal Processing Letters.

[13]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[14]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[15]  Bernard P. Zeigler,et al.  Theory of Modeling and Simulation: Integrating Discrete Event and Continuous Complex Dynamic Systems , 2000 .

[16]  David Haussler,et al.  Convolution kernels on discrete structures , 1999 .

[17]  R. Schaback.  A unified theory of radial basis functions Native Hilbert spaces for radial basis functions II , 2000 .

[18]  Bernhard Schölkopf,et al.  Semiparametric Support Vector and Linear Programming Machines , 1998, NIPS.

[19]  J. Mercer.  Functions of Positive and Negative Type, and their Connection with the Theory of Integral Equations , 1909 .

[20]  A. Berlinet,et al.  Reproducing kernel Hilbert spaces in probability and statistics , 2004 .

[21]  Mohamed Mohandes,et al.  Support vector machines for wind speed prediction , 2004 .

[22]  Olvi L. Mangasarian,et al.  Nonlinear Knowledge in Kernel Approximation , 2007, IEEE Transactions on Neural Networks.

[23]  Paul K. Davis,et al.  Preface , 2021 .

[24]  G. Wahba.  Support vector machines, reproducing kernel Hilbert spaces, and randomized GACV , 1999 .

[25]  Joachim Stöckler.  Multivariate Bernoulli splines and the periodic interpolation problem , 1991 .

[26]  Xin Zhao,et al.  A New Composition Method of Admissible Support Vector Kernel Based on Reproducing Kernel , 2010 .

[27]  Gérard Bloch,et al.  Incorporating prior knowledge in support vector regression , 2007, Machine Learning.

[28]  S. S. Chaudhry *,et al.  Application of genetic algorithms in production and operations management: a review , 2005 .

[29]  Jude W. Shavlik,et al.  Knowledge-Based Kernel Approximation , 2004, J. Mach. Learn. Res..

[31]  Gérard Bloch,et al.  Support vector regression from simulation data and few experimental samples , 2008, Inf. Sci..