On kernel design for regularized LTI system identification

Abstract There are two key issues for the kernel-based regularization method: one is how to design a suitable kernel to embed in the kernel the prior knowledge of the LTI system to be identified, and the other one is how to tune the kernel such that the resulting regularized impulse response estimator can achieve a good bias–variance tradeoff. In this paper, we focus on the issue of kernel design. Depending on the type of the prior knowledge, we propose two methods to design kernels: one is from a machine learning perspective and the other one is from a system theory perspective. We also provide analysis results for both methods, which not only enhances our understanding for the existing kernels but also directs the design of new kernels.

[1]  Richard A. Silverman,et al.  Locally stationary random processes , 2018, IRE Trans. Inf. Theory.

[2]  Lennart Ljung,et al.  Maximum entropy properties of discrete-time first-order stable spline kernel , 2014, Autom..

[3]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[4]  Biao Huang,et al.  System Identification , 2000, Control Theory for Physicists.

[5]  Tianshi Chen,et al.  On Input Design for Regularized LTI System Identification: Power-constrained Input , 2017, Autom..

[6]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[7]  Lennart Ljung,et al.  Comparing different approaches to model error modeling in robust identification , 2002, Autom..

[8]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[9]  Tianshi Chen,et al.  Continuous-Time DC Kernel—A Stable Generalized First-Order Spline Kernel , 2018, IEEE Transactions on Automatic Control.

[10]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[11]  Lennart Ljung,et al.  Model Validation and Model Error Modeling , 1999 .

[12]  Henrik Ohlsson,et al.  On the estimation of transfer functions, regularizations and Gaussian processes - Revisited , 2012, Autom..

[13]  HighWire Press.  Philosophical transactions of the Royal Society of London. Series A, Containing papers of a mathematical or physical character , 1896 .

[14]  V. Hutson.  Integral Equations , 1967, Nature.

[15]  Francesca P. Carli.  On the maximum entropy property of the first-order stable spline kernel and its implications , 2014, 2014 IEEE Conference on Control Applications (CCA).

[16]  C. Carmeli,et al.  VECTOR VALUED REPRODUCING KERNEL HILBERT SPACES OF INTEGRABLE FUNCTIONS AND MERCER THEOREM , 2006 .

[17]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[18]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[19]  Doreen Meier,et al.  Introduction To Stochastic Control Theory , 2016 .

[20]  Takuya Kon-no,et al.  Transactions of the American Mathematical Society , 1996 .

[21]  L. Ljung,et al.  Constructive state space model induced kernels for regularized system identification , 2014 .

[22]  Lennart Ljung,et al.  On the design of multiple kernels for nonparametric linear system identification , 2014, 53rd IEEE Conference on Decision and Control.

[23]  Graham C. Goodwin,et al.  Estimation of model quality , 1994, Autom..

[24]  Robert Sitton,et al.  New York and London , 2014 .

[25]  E. Jaynes.  On the rationale of maximum-entropy methods , 1982, Proceedings of the IEEE.

[26]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[27]  Lennart Ljung,et al.  System Identification Via Sparse Multiple Kernel-Based Regularization Using Sequential Convex Optimization Techniques , 2014, IEEE Transactions on Automatic Control.

[28]  Francesco Dinuzzo,et al.  Kernels for Linear Time Invariant System Identification , 2012, SIAM J. Control. Optim..

[29]  Giuseppe De Nicolao,et al.  A new kernel-based approach for linear system identification , 2010, Autom..

[30]  A. Yaglom.  Correlation Theory of Stationary and Related Random Functions I: Basic Results , 1987 .

[31]  Graham C. Goodwin,et al.  Estimated Transfer Functions with Application to Model Order Selection , 1992 .

[32]  Alessandro Chiuso,et al.  Tuning complexity in regularized kernel-based regression and linear system identification: The robustness of the marginal likelihood estimator , 2015, Autom..

[33]  Graham C. Goodwin,et al.  Non-stationary stochastic embedding for transfer function estimation , 1999, Autom..

[34]  Chi-Tsong Chen,et al.  Linear System Theory and Design , 1995 .

[35]  Alessandro Chiuso,et al.  Prediction error identification of linear systems: A nonparametric Gaussian regression approach , 2011, Autom..

[36]  Francesca P. Carli,et al.  Maximum Entropy Kernels for System Identification , 2014, IEEE Transactions on Automatic Control.

[37]  L. Ljung,et al.  On kernel structures for regularized system identification (II): a system theory perspective , 2015 .

[38]  Hongwei Sun,et al.  Mercer theorem for RKHS on noncompact sets , 2005, J. Complex..

[39]  Mi-Ching Tsai,et al.  Robust and Optimal Control , 2014 .

[40]  Alessandro Chiuso,et al.  Regularization and Bayesian learning in dynamical systems: Past, present and future , 2015, Annu. Rev. Control..

[41]  Bernhard Schölkopf,et al.  Regularization Networks and Support Vector Machines , 2000 .

[42]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[43]  Lennart Ljung,et al.  Tuning of Hyperparameters for FIR models - an Asymptotic Theory , 2017 .

[44]  Marc G. Genton,et al.  Classes of Kernels for Machine Learning: A Statistics Perspective , 2002, J. Mach. Learn. Res..

[45]  Lennart Ljung,et al.  On Asymptotic Properties of Hyperparameter Estimators for Kernel-based Regularization Methods , 2017, Autom..

[46]  L. Ljung,et al.  Control theory : multivariable and nonlinear methods , 2000 .

[47]  T. McKelveyDepartment.  On the Use of Regularization in System Identification , 1992 .

[48]  Lennart Ljung,et al.  Kernel methods in system identification, machine learning and function estimation: A survey , 2014, Autom..

[49]  Johan Schoukens,et al.  Filter-based regularisation for impulse response modelling , 2016, ArXiv.

[50]  Thomas B. Schön,et al.  System identification of nonlinear state-space models , 2011, Autom..

[51]  Lennart Ljung,et al.  Implementation of algorithms for tuning parameters in regularized least squares problems in system identification , 2013, Autom..