Structure detection and parameter estimation for NARX models in a unified EM framework

In this paper, we consider structure detection and parameter estimation of the nonlinear auto-regressive with exogenous inputs (NARX) model, using the EM (expectation-maximisation) algorithm. The parameter estimation step uses particle smoothing to obtain the necessary expectations in the E-step and the parameters are then estimated in closed form in the M-step. The model structure detection is performed using an F-test, which makes use of the parameter information matrix (inverse of the covariance matrix), obtained from an augmentation of the EM algorithm. The steps for obtaining the information matrix are robust, guaranteeing a positive semi-definite information matrix to use in the structure detection step. For the case of unknown model orders, a method is proposed using the stochastic complexity (SC) information criterion for selecting between candidate models. The SC is composed of the information matrix (representing model complexity) and a likelihood estimate (representing model accuracy), which are both generated as byproducts of the augmented EM algorithm. Numerical results demonstrate that the EM approach performs well in comparison to a standard alternative based on orthogonal least squares, and also avoids the need to estimate a noise model for the case of measurement noise corrupted output signals.

[1]  James P. Reilly,et al.  An EM Algorithm for Nonlinear State Estimation With Model Uncertainties , 2008, IEEE Transactions on Signal Processing.

[2]  S. Roweis,et al.  Learning Nonlinear Dynamical Systems Using the Expectation–Maximization Algorithm , 2001 .

[3]  A. Doucet,et al.  A Tutorial on Particle Filtering and Smoothing: Fifteen years later , 2008 .

[4]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[5]  S. Billings,et al.  Orthogonal parameter estimation algorithm for non-linear stochastic systems , 1988 .

[6]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[7]  I. J. Leontaritis,et al.  Input-output parametric models for non-linear systems Part II: stochastic non-linear systems , 1985 .

[8]  Steve A. Billings,et al.  Term and variable selection for non-linear system identification , 2004 .

[9]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[10]  Jin-Chuan Duan,et al.  A stable estimator of the information matrix under EM for dependent data , 2011, Stat. Comput..

[11]  T. Louis.  Finding the Observed Information Matrix When Using the EM Algorithm , 1982 .

[12]  R. Bhushan Gopaluni,et al.  Nonlinear system identification under missing observations: The case of unknown model structure , 2010 .

[13]  David S. Stoffer,et al.  Time series analysis and its applications , 2000 .

[14]  L. Piroddi,et al.  An identification algorithm for polynomial NARX models based on simulation error minimization , 2003 .

[15]  Giorgio Battistelli,et al.  Unfalsified Virtual Reference Adaptive Switching Control of Plants with Persistent Disturbances , 2008 .

[16]  Alf Isaksson,et al.  Identification of ARX-models subject to missing data , 1993, IEEE Trans. Autom. Control..

[17]  R. Engle.  Wald, likelihood ratio, and Lagrange multiplier tests in econometrics , 1984 .

[18]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[19]  Brett Ninness,et al.  Robust maximum-likelihood estimation of multivariable dynamic systems , 2005, Autom..

[20]  S. Billings,et al.  Algorithms for minimal model structure detection in nonlinear dynamic system identification , 1997 .

[21]  Thomas B. Schön,et al.  Parameter Estimation for Discrete-Time Nonlinear Systems Using EM , 2008 .

[22]  Kang Li,et al.  A two-stage algorithm for identification of nonlinear dynamic systems , 2006, Autom..

[23]  Sean R. Anderson,et al.  Maximum-Likelihood Estimation of Delta-Domain Model Parameters From Noisy Output Signals , 2008, IEEE Transactions on Signal Processing.

[24]  W. Newey,et al.  Large sample estimation and hypothesis testing , 1986 .

[25]  Thomas Bo Schön,et al.  Maximum Likelihood Nonlinear System Estimation , 2006 .

[26]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[27]  R. B. Gopaluni.  A particle filter approach to identification of nonlinear processes under missing observations , 2008 .

[28]  S. Billings,et al.  A prediction-error and stepwise-regression estimation algorithm for non-linear systems , 1986 .

[29]  H. Akaike.  A new look at the statistical model identification , 1974 .

[30]  Bin Yu,et al.  Model Selection and the Principle of Minimum Description Length , 2001 .

[31]  Bart De Moor,et al.  Subspace Identification for Linear Systems: Theory ― Implementation ― Applications , 2011 .

[32]  Visakan Kadirkamanathan,et al.  Particle filtering-based fault detection in non-linear stochastic systems , 2002, Int. J. Syst. Sci..

[33]  Thomas B. Schön,et al.  Particle Filter Approach to Nonlinear System Identification under Missing Observations with a Real Application , 2009 .

[34]  Simon J. Godsill,et al.  On sequential Monte Carlo sampling methods for Bayesian filtering , 2000, Stat. Comput..

[35]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[36]  R. Kearney,et al.  A bootstrap method for structure detection of NARMAX models , 2004 .

[37]  Thomas B. Schön,et al.  System identification of nonlinear state-space models , 2011, Autom..

[38]  Jorma Rissanen,et al.  Stochastic Complexity in Statistical Inquiry , 1989, World Scientific Series in Computer Science.

[39]  Petar M. Djuric,et al.  Asymptotic MAP criteria for model selection , 1998, IEEE Trans. Signal Process..

[40]  W. Newey,et al.  A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelationconsistent Covariance Matrix , 1986 .

[41]  R. Shumway,et al.  AN APPROACH TO TIME SERIES SMOOTHING AND FORECASTING USING THE EM ALGORITHM , 1982 .

[42]  Heinz Unbehauen,et al.  Structure identification of nonlinear dynamic systems - A survey on input/output approaches , 1990, Autom..

[43]  V. Kadirkamanathan,et al.  Modelling and identification of non-linear deterministic systems in the delta-domain , 2007, Autom..

[44]  New York Dover,et al.  ON THE CONVERGENCE PROPERTIES OF THE EM ALGORITHM , 1983 .

[45]  Christophe Andrieu,et al.  Particle methods for change detection, system identification, and control , 2004, Proceedings of the IEEE.