Model structure selection using an integrated forward orthogonal search algorithm assisted by squared correlation and mutual information

Model structure selection plays a key role in non-linear system identification. The first step in non-linear system identification is to determine which model terms should be included in the model. Once significant model terms have been determined, a model selection criterion can then be applied to select a suitable model subset. The well known Orthogonal Least Squares (OLS) type algorithms are one of the most efficient and commonly used techniques for model structure selection. However, it has been observed that the OLS type algorithms may occasionally select incorrect model terms or yield a redundant model subset in the presence of particular noise structures or input signals. A very efficient Integrated Forward Orthogonal Search (IFOS) algorithm, which is assisted by the squared correlation and mutual information, and which incorporates a Generalised Cross-Validation (GCV) criterion and hypothesis tests, is introduced to overcome these limitations in model structure selection.

[1]  Alan J. Miller.  Subset Selection in Regression , 1992 .

[2]  David M. Allen,et al.  The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction , 1974 .

[3]  William J. Fitzgerald,et al.  Parameter-based hypothesis tests for model selection , 1995, Signal Process..

[4]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[5]  Sheng Chen,et al.  Sparse kernel regression modeling using combined locally regularized orthogonal least squares and D-optimality experimental design , 2003, IEEE Trans. Autom. Control..

[6]  D. Freedman,et al.  How Many Variables Should Be Entered in a Regression Equation , 1983 .

[7]  S. A. Billings,et al.  The wavelet-NARMAX representation: A hybrid model structure combining polynomial models with multiresolution wavelet decompositions , 2005, Int. J. Syst. Sci..

[8]  Stephen A. Billings,et al.  A new class of wavelet networks for nonlinear system identification , 2005, IEEE Transactions on Neural Networks.

[9]  Stephen A. Billings,et al.  Radial Basis Function Network Configuration Using Mutual Information and the Orthogonal Least Squares Algorithm , 1996, Neural Networks.

[10]  R. Savit,et al.  Time series and dependent variables , 1991 .

[11]  L. Breiman.  Better subset regression using the nonnegative garrote , 1995 .

[12]  R. Moddemeijer.  On estimation of entropy and mutual information of continuous distributions , 1989 .

[13]  Zhifeng Zhang,et al.  Adaptive time-frequency decompositions , 1994 .

[14]  Xia Hong,et al.  Nonlinear model structure design and construction using orthogonal least squares and D-optimality design , 2002, IEEE Trans. Neural Networks.

[15]  Alan J. Miller,et al.  Subset Selection in Regression , 1991 .

[16]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[17]  Roberto Battiti,et al.  Using mutual information for selecting features in supervised neural net learning , 1994, IEEE Trans. Neural Networks.

[18]  Ingela Lind.  REGRESSOR SELECTION WITH THE ANALYSIS OF VARIANCE METHOD , 2002 .

[19]  John E. Moody,et al.  The Effective Number of Parameters: An Analysis of Generalization and Regularization in Nonlinear Learning Systems , 1991, NIPS.

[20]  X. X. Wang,et al.  Sparse incremental regression modeling using correlation criterion with boosting search , 2005, IEEE Signal Processing Letters.

[21]  Stephen A. Billings,et al.  Global analysis and model validation in nonlinear system identification , 1994, Nonlinear Dynamics.

[22]  Steve A. Billings,et al.  Term and variable selection for non-linear system identification , 2004 .

[23]  Jammalamadaka.  Introduction to Linear Regression Analysis (3rd ed.) , 2003 .

[24]  Y. Selen,et al.  Model-order selection: a review of information criterion rules , 2004, IEEE Signal Processing Magazine.

[25]  Rudy Moddemeijer,et al.  A statistic to estimate the variance of the histogram-based mutual information estimator based on dependent pairs of observations , 1999, Signal Process..

[26]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[27]  Sheng Chen,et al.  Identification of MIMO non-linear systems using a forward-regression orthogonal estimator , 1989 .

[28]  Luis A. Aguirre,et al.  Nonlinearities in NARX polynomial models: representation and estimation , 2002 .

[29]  Petre Stoica,et al.  Decentralized Control , 2018, The Control Systems Handbook.

[30]  S. Billings,et al.  Algorithms for minimal model structure detection in nonlinear dynamic system identification , 1997 .

[31]  L. A. Aguirre,et al.  Improved structure selection for nonlinear models based on term clustering , 1995 .

[32]  Stephen A. Billings,et al.  The Determination of Multivariable Nonlinear Models for Dynamic Systems Using neural Networks , 1996 .

[33]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[34]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[35]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[36]  Sheng Chen,et al.  M-estimator and D-optimality model construction using orthogonal forward regression , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[37]  H. Akaike.  A new look at the statistical model identification , 1974 .

[38]  L. A. Aguirre,et al.  EFFECTS OF THE SAMPLING TIME ON THE DYNAMICS AND IDENTIFICATION OF NONLINEAR MODELS , 1995 .

[39]  Igor Vajda,et al.  Estimation of the Information by an Adaptive Partitioning of the Observation Space , 1999, IEEE Trans. Inf. Theory.

[40]  Ferenc Szeifert,et al.  Genetic programming for the identification of nonlinear input-output models , 2005 .

[41]  Elizabeth A. Peck,et al.  Introduction to Linear Regression Analysis , 2001 .

[42]  R. R. Hocking.  Developments in linear regression methodology: 1959-1982 , 1983 .

[43]  Stephen A. Billings,et al.  of overparametrization in non-linear system identification and neural networks , 2002, Int. J. Syst. Sci..

[44]  L. Piroddi,et al.  An identification algorithm for polynomial NARX models based on simulation error minimization , 2003 .

[45]  Torsten Söderström,et al.  Model-structure selection by cross-validation , 1986 .

[46]  Sheng Chen,et al.  Regularized orthogonal least squares algorithm for constructing radial basis function networks , 1996 .

[47]  G. L. Zheng,et al.  Qualitative validation and generalization in non-linear system identification , 1999 .

[48]  Jianhua Z. Huang,et al.  Identification of non‐linear additive autoregressive models , 2004 .

[49]  J. Abonyi,et al.  Model Order Selection of Nonlinear Input-Output Models - A Clustering Based Approach , 2004 .

[50]  L. A. Aguirre,et al.  Dynamical effects of overparametrization in nonlinear models , 1995 .

[51]  Gene H. Golub,et al.  Generalized cross-validation as a method for choosing a good ridge parameter , 1979, Milestones in Matrix Computation.

[52]  S. Billings,et al.  Fast orthogonal identification of nonlinear stochastic models and radial basis function neural networks , 1996 .

[53]  Stephen A. Billings,et al.  Sparse Model Identification Using a Forward Orthogonal Regression Algorithm Aided by Mutual Information , 2007, IEEE Transactions on Neural Networks.

[54]  Mark J. L. Orr.  Optimising the widths of radial basis functions , 1998, Proceedings 5th Brazilian Symposium on Neural Networks (Cat. No.98EX209).

[55]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[56]  J. Shao.  Linear Model Selection by Cross-validation , 1993 .

[57]  Stephen A. Billings,et al.  An adaptive orthogonal search algorithm for model subset selection and non-linear system identification , 2008, Int. J. Control.

[58]  Xia Hong,et al.  Adaptive Modelling, Estimation and Fusion from Data: A Neurofuzzy Approach , 2002, Advanced information processing.

[59]  P. Vieu.  Order Choice in Nonlinear Autoregressive Models , 1995 .

[60]  Heinz Unbehauen,et al.  Structure identification of nonlinear dynamic systems - A survey on input/output approaches , 1990, Autom..

[61]  Stephen A. Billings,et al.  RETRIEVING DYNAMICAL INVARIANTS FROM CHAOTIC DATA USING NARMAX MODELS , 1995 .

[62]  S. Billings,et al.  Orthogonal parameter estimation algorithm for non-linear stochastic systems , 1988 .

[63]  Dale E. Seborg,et al.  Determination of model order for NARX models directly from input-output data , 1998 .

[64]  I. J. Leontaritis,et al.  Input-output parametric models for non-linear systems Part II: stochastic non-linear systems , 1985 .

[65]  Liam Paninski,et al.  Estimation of Entropy and Mutual Information , 2003, Neural Computation.

[66]  Mark J. L. Orr,et al.  Regularization in the Selection of Radial Basis Function Centers , 1995, Neural Computation.

[67]  Dag Tjøstheim,et al.  Nonparametric Identification of Nonlinear Time Series: Projections , 1994 .

[68]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[69]  L. A. Aguirre,et al.  Use of a priori information in the identification of global nonlinear models-a case study using a buck converter , 2000 .

[70]  Timo Teräsvirta,et al.  A SIMPLE VARIABLE SELECTION TECHNIQUE FOR NONLINEAR MODELS , 2001 .

[71]  S. A. Billings,et al.  The identification of linear and non-linear models of a turbocharged automotive diesel engine , 1989 .

[72]  R. R. Hocking.  The analysis and selection of variables in linear regression , 1976 .

[73]  L. A. Aguirre,et al.  Imposing steady-state performance on identified nonlinear polynomial models by means of constrained parameter estimation , 2004 .

[74]  R. Pearson.  Discrete-time Dynamic Models , 1999 .

[75]  J. Friedman,et al.  FLEXIBLE PARSIMONIOUS SMOOTHING AND ADDITIVE MODELING , 1989 .

[76]  L. Ljung,et al.  Overtraining, regularization and searching for a minimum, with application to neural networks , 1995 .

[77]  Sheng Chen,et al.  Identification of non-linear output-affine systems using an orthogonal least-squares algorithm , 1988 .

[78]  R. R. Hocking.  Developments in Linear Regression Methodology: 1959–l982 , 1983 .

[79]  O. Nelles.  Nonlinear System Identification , 2001 .

[80]  A. Barron,et al.  Discussion: Multivariate Adaptive Regression Splines , 1991 .

[81]  O. Nelles.  Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models , 2000 .

[82]  Ulrich Anders,et al.  Model selection in neural networks , 1999, Neural Networks.

[83]  S. A. Billings,et al.  Experimental design and identifiability for non-linear systems , 1987 .

[84]  L. A. Aguirre,et al.  Validating Identified Nonlinear Models with Chaotic Dynamics , 1994 .