A new kernel-based approach for linear system identification

This paper describes a new kernel-based approach for linear system identification of stable systems. We model the impulse response as the realization of a Gaussian process whose statistics, differently from previously adopted priors, include information not only on smoothness but also on BIBO-stability. The associated autocovariance defines what we call a stable spline kernel. The corresponding minimum variance estimate belongs to a reproducing kernel Hilbert space which is spectrally characterized. Compared to parametric identification techniques, the impulse response of the system is searched for within an infinite-dimensional space, dense in the space of continuous functions. Overparametrization is avoided by tuning few hyperparameters via marginal likelihood maximization. The proposed approach may prove particularly useful in the context of robust identification in order to obtain reduced order models by exploiting a two-step procedure that projects the nonparametric estimate onto the space of nominal models. The continuous-time derivation immediately extends to the discrete-time case. On several continuous- and discrete-time benchmarks taken from the literature the proposed approach compares very favorably with the existing parametric and nonparametric techniques.

[1]  L. Ljung,et al.  Design variables for bias distribution in transfer function estimation , 1986, The 23rd IEEE Conference on Decision and Control.

[2]  J. Norton,et al.  Bounding Approaches to System Identification , 1996 .

[3]  P.M. Mäkilä,et al.  Worst-case control-relevant identification , 1995, Autom..

[4]  Gianluigi Pillonetto,et al.  Bayes and empirical Bayes semi-blind deconvolution using eigenfunctions of a prior covariance , 2007, Autom..

[5]  Tor Arne Johansen,et al.  On Tikhonov regularization, bias and variance in nonlinear system identification , 1997, Autom..

[6]  G. Wahba.  Support vector machines, reproducing kernel Hilbert spaces, and randomized GACV , 1999 .

[7]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[8]  Christopher M. Bishop,et al.  Neural networks and machine learning , 1998 .

[9]  Alessandro Chiuso,et al.  Predictor estimation via Gaussian regression , 2008, 2008 47th IEEE Conference on Decision and Control.

[10]  Giuseppe De Nicolao,et al.  Nonparametric identification of population models via Gaussian processes , 2007, Autom..

[11]  Hongwei Sun,et al.  Mercer theorem for RKHS on noncompact sets , 2005, J. Complex..

[12]  Mathukumalli Vidyasagar,et al.  A Theory of Learning and Generalization , 1997 .

[13]  Huaiyu Zhu,et al.  Bayesian regression filters and the issue of priors , 1996, Neural Computing & Applications.

[14]  G. Pillonetto,et al.  Estimating parameters and stochastic functions of one variable using nonlinear measurement models , 2004 .

[15]  Mario Milanese,et al.  H∞ identification and model quality evaluation , 1997, IEEE Trans. Autom. Control..

[16]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[17]  Antonio Vicino,et al.  Optimal estimation theory for dynamic systems with set membership uncertainty: An overview , 1991, Autom..

[18]  P. V. D. Hof,et al.  Identification of probabilistic system uncertainty regions by explicit evaluation of bias and variance errors , 1997, IEEE Trans. Autom. Control..

[19]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[20]  Gianluigi Pillonetto,et al.  Input estimation in nonlinear dynamical systems using differential algebra techniques , 2006, Autom..

[21]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[22]  Carl E. Rasmussen,et al.  In Advances in Neural Information Processing Systems , 2011 .

[23]  Manfred Opper,et al.  Finite-Dimensional Approximation of Gaussian Processes , 1998, NIPS.

[24]  R. Tapia,et al.  Nonparametric Function Estimation, Modeling, and Simulation , 1987 .

[25]  Graham C. Goodwin,et al.  Non-stationary stochastic embedding for transfer function estimation , 1999, Autom..

[26]  Andrea Garulli,et al.  Conditional central algorithms for worst case set-membership identification and filtering , 2000, IEEE Trans. Autom. Control..

[27]  MethodsDavid,et al.  Nonparametric Bayesian Regression , 1998 .

[28]  T. Poggio,et al.  Networks and the best approximation property , 1990, Biological Cybernetics.

[29]  Fredrik Tjärnström,et al.  A Nonparametric Approach to Model Error Modeling , 2000 .

[30]  Alessandro Chiuso,et al.  Subspace identification using predictor estimation via Gaussian regression , 2008, 2008 47th IEEE Conference on Decision and Control.

[31]  V. Burenkov.  Sobolev spaces on domains , 1998 .

[32]  Michel Gevers,et al.  Explicit expression of the parameter bias in identification of Laguerre models from step responses , 2004, Syst. Control. Lett..

[33]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[34]  D. Freedman.  On the Bernstein-von Mises Theorem with Infinite Dimensional Parameters , 1999 .

[35]  Graham C. Goodwin,et al.  Non-stationary stochastic embedding for transfer function estimation , 1999, Autom..

[36]  Håkan Hjalmarsson,et al.  From experiment design to closed-loop control , 2005, Autom..

[37]  R. Shah,et al.  Least Squares Support Vector Machines , 2022 .

[38]  Graham C. Goodwin,et al.  Estimated Transfer Functions with Application to Model Order Selection , 1992 .

[39]  Biao Huang,et al.  System Identification , 2000, Control Theory for Physicists.

[40]  Witold Pedrycz,et al.  Bounding approaches to system identification , 1997 .

[41]  Lennart Ljung,et al.  Comparing different approaches to model error modeling in robust identification , 2002, Autom..

[42]  H. Hjalmarsson,et al.  Composite modeling of transfer functions , 1995, Proceedings of 1995 34th IEEE Conference on Decision and Control.

[43]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[44]  Lennart Ljung,et al.  Model Validation and Model Error Modeling , 1999 .

[45]  L. Marton,et al.  Advances in Electronics and Electron Physics , 1958 .

[46]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[47]  Petros G. Voulgaris,et al.  On optimal ℓ∞ to ℓ∞ filtering , 1995, Autom..

[48]  Bernhard Schölkopf,et al.  Bayesian Kernel Methods , 2002, Machine Learning Summer School.

[49]  G. Wahba.  Spline models for observational data , 1990 .

[50]  S. Smale,et al.  Learning Theory Estimates via Integral Operators and Their Approximations , 2007 .

[51]  Milan Lukić,et al.  Stochastic processes with sample paths in reproducing kernel Hilbert spaces , 2001 .

[52]  G. Wahba.  Practical Approximate Solutions to Linear Operator Equations When the Data are Noisy , 1977 .

[53]  Christopher K. I. Williams,et al.  Gaussian regression and optimal finite dimensional linear models , 1997 .

[54]  M. Bertero.  Linear Inverse and III-Posed Problems , 1989 .