Prediction error identification of linear systems: A nonparametric Gaussian regression approach

A novel Bayesian paradigm for the identification of output error models has recently been proposed in which, in place of postulating finite-dimensional models of the system transfer function, the system impulse response is searched for within an infinite-dimensional space. In this paper, such a nonparametric approach is applied to the design of optimal predictors and discrete-time models based on prediction error minimization by interpreting the predictor impulse responses as realizations of Gaussian processes. The proposed scheme describes the predictor impulse responses as the convolution of an infinite-dimensional response with a low-dimensional parametric response that captures possible high-frequency dynamics. Overparameterization is avoided because the model involves only a few hyperparameters that are tuned via marginal likelihood maximization. Numerical experiments, with data generated by ARMAX and infinite-dimensional models, show the definite advantages of the new approach over standard parametric prediction error techniques and subspace methods both in terms of predictive capability on new data and accuracy in reconstruction of system impulse responses.

[1]  David J. C. MacKay,et al.  Comparison of Approximate Methods for Handling Hyperparameters , 1999, Neural Computation.

[2]  David M. Auslander,et al.  Control and dynamic systems , 1970 .

[3]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[4]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[5]  Bernhard Schölkopf,et al.  Bayesian Kernel Methods , 2002, Machine Learning Summer School.

[6]  Javad Mohammadpour,et al.  Control of linear parameter varying systems with applications , 2012 .

[7]  G. Wahba.  Spline models for observational data , 1990 .

[8]  Michael Athans,et al.  Analysis of gain scheduled control for nonlinear plants , 1990 .

[9]  S. Smale,et al.  Learning Theory Estimates via Integral Operators and Their Approximations , 2007 .

[10]  H. Akaike.  A new look at the statistical model identification , 1974 .

[11]  G. Pillonetto,et al.  Nonlinear system identification via Gaussian regression and mixtures of kernels , 2009 .

[12]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[13]  Antonio Vicino,et al.  Optimal estimation theory for dynamic systems with set membership uncertainty: An overview , 1991, Autom..

[14]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[15]  Peter C. Young,et al.  Recursive and en-bloc approaches to signal extraction , 1999 .

[16]  B. Ninness,et al.  System identification of linear parameter varying state-space models , 2011 .

[17]  W. Gersch,et al.  A smoothness priors long AR model method for spectral estimation , 1985, IEEE Transactions on Automatic Control.

[18]  Andrew Harvey,et al.  Forecasting, Structural Time Series Models and the Kalman Filter , 1990 .

[19]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[20]  Henrik Ohlsson,et al.  On the estimation of transfer functions, regularizations and Gaussian processes - Revisited , 2012, Autom..

[21]  U. Desai,et al.  A realization approach to stochastic model reduction , 1985 .

[22]  Gianluigi Pillonetto,et al.  Input estimation in nonlinear dynamical systems using differential algebra techniques , 2006, Autom..

[23]  L. Silverman,et al.  Model reduction via balanced state space representations , 1982 .

[24]  Hugues Garnier,et al.  Refined instrumental variable methods for identification of LPV Box-Jenkins models , 2010, Autom..

[25]  P. Young,et al.  Identification of non-linear stochastic systems by state dependent parameter estimation , 2001 .

[26]  M. Bertero.  Linear Inverse and III-Posed Problems , 1989 .

[27]  Giuseppe De Nicolao,et al.  Regularization networks: fast weight calculation via Kalman filtering , 2001, IEEE Trans. Neural Networks.

[28]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[29]  B. M. Pötscher,et al.  MODEL SELECTION AND INFERENCE: FACTS AND FICTION , 2005, Econometric Theory.

[30]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[31]  T. Louis,et al.  Bayes and Empirical Bayes Methods for Data Analysis. , 1997 .

[32]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[33]  U. Desai,et al.  A transformation approach to stochastic model reduction , 1984 .

[34]  Roland Tóth,et al.  Asymptotically optimal orthonormal basis functions for LPV system identification , 2009, Autom..

[35]  G. Kitagawa.  Smoothness priors analysis of time series , 1996 .

[36]  G. Kitagawa,et al.  A Smoothness Priors–State Space Modeling of Time Series with Trend and Seasonality , 1984 .

[37]  K. Mengersen,et al.  A Bayesian – Decision Theoretic Approach to Model Error Modeling , 2022 .

[38]  Bassam Bamieh,et al.  Identification of linear parameter varying models , 2002 .

[39]  Giuseppe De Nicolao,et al.  Fast algorithms for nonparametric population modeling of large data sets , 2009, Autom..

[40]  Lloyd N. Trefethen,et al.  Barycentric Lagrange Interpolation , 2004, SIAM Rev..

[41]  Roland Toth,et al.  Modeling and Identification of Linear Parameter-Varying Systems , 2010 .

[42]  Yuhong Yang.  Can the Strengths of AIC and BIC Be Shared , 2005 .

[43]  E. Hannan.  The Estimation of the Order of an ARMA Process , 1980 .

[44]  Alessandro Chiuso,et al.  Subspace identification using predictor estimation via Gaussian regression , 2008, 2008 47th IEEE Conference on Decision and Control.

[45]  D. Barry.  Nonparametric Bayesian Regression , 1986 .

[46]  Gianluigi Pillonetto,et al.  Bayes and empirical Bayes semi-blind deconvolution using eigenfunctions of a prior covariance , 2007, Autom..

[47]  Okko H. Bosgra,et al.  LPV control for a wafer stage: beyond the theoretical solution , 2005 .

[48]  Gene H. Golub,et al.  Generalized cross-validation as a method for choosing a good ridge parameter , 1979, Milestones in Matrix Computation.

[49]  Giuseppe De Nicolao,et al.  Nonparametric identification of population models via Gaussian processes , 2007, Autom..

[50]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[51]  Dietmar Bauer,et al.  Order estimation for subspace methods , 2001, Autom..

[52]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[53]  Alessandro Chiuso,et al.  The role of vector autoregressive modeling in predictor-based subspace identification , 2007, Autom..

[54]  Howard L. Weinert.  Fixed Interval Smoothing for State Space Models , 2001 .

[55]  Liuping Wang,et al.  Use of PRESS residuals in dynamic system identification , 1996, Autom..

[56]  Graham C. Goodwin,et al.  Non-stationary stochastic embedding for transfer function estimation , 1999, Autom..

[57]  N. D. Bruijn.  Asymptotic methods in analysis , 1958 .

[58]  G. Pillonetto,et al.  Estimating parameters and stochastic functions of one variable using nonlinear measurement models , 2004 .

[59]  Michel Loève,et al.  Probability Theory I , 1977 .

[60]  J. Suykens,et al.  Subspace identification of Hammerstein systems using least squares support vector machines , 2005 .

[61]  Peter C. Young,et al.  Recursive Estimation, Forecasting, and Adaptive Control , 1989 .

[62]  Michel Verhaegen,et al.  Subspace identification of Bilinear and LPV systems for open- and closed-loop data , 2009, Autom..

[63]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[64]  Bonaventure Intercontinental,et al.  ON DECISION AND CONTROL , 1985 .

[65]  Alessandro Chiuso,et al.  A New Kernel-Based Approach for NonlinearSystem Identification , 2011, IEEE Transactions on Automatic Control.

[66]  Robert B. Litterman,et al.  Forecasting and Conditional Projection Using Realistic Prior Distributions , 1983 .

[67]  Dennis V. Lindley,et al.  Empirical Bayes Methods , 1974 .

[68]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[69]  Giuseppe De Nicolao,et al.  A new kernel-based approach for linear system identification , 2010, Autom..

[70]  Mario Sznaier,et al.  An LMI approach to control-oriented identification and model (In) validation of LPV systems , 2003, IEEE Trans. Autom. Control..

[71]  Petre Stoica,et al.  Decentralized Control , 2018, The Control Systems Handbook.

[72]  Paul M.J. Van den Hof,et al.  Prediction-Error Identification of LPV Systems: Present and Beyond , 2012 .