Non-Linear System Identification with Neural Networks

This thesis addresses the non-linear system identification problem, and in particular, investigates the use of neural networks in system identification. An overview of different possible mode! structures is given in a common framework. A nonlinear structure is described as the concatenation of a map from the observed data to the regressor, and a map from the regressor to the output space. This divides the model structure selection problem into two problems with lower complexity: that of choosing the regressor and that of choosing the non-linear map.The possible choices for the regressors consists of past inputs and outputs, and filtered versions of them. The dynamics of the mode! depends on the choice of regressor, and families of different mode! structures are suggested based on analogies to linear black-box models. State-space models are also described within this common framework by a special choice of regressor. It is shown that state-space models which have no parameters in the state update function can be viewed as an input-output mode! preceded by a pre-filter. A parameterized state update function, on the other hand, can be seen as a data driven regressor selector. The second step of the nonlinear identification is the mapping from the regressor to the output space. It is often advantageous to try some intermediate mappings between the linear and the general non-linear mapping. Such non-linear black-box mappings are discussed and motivated by considering different noise assumptions.The validation of a linear mode! should contain a test for non-linearities and it is shown that, in general, it is easy to detect non-linearities. This implies that it is not worth spending too much energy searching for optimal non-linear validation methods for a specific problem. lnstead the validation method should be chosen so that it is easy to apply. Two such methods, based on polynomials and neural nets, are suggested. Further, two validation methods, the correlation-test and the parametric F-test, are investigated. It is shown that under certain conditions these methodscoincide.Parameter estimates are usually based on criterion minimization. In connection with neural nets it has been noted that it is not always optimal to try to find the absolute minimum point of the criterion. Instead a better estimate can be obtained if the numerical search for the minimum is prematurely stopped. A forma! connection between this stopped search and regularization is given. It is shown that the numerical minimization of the criterion can be view as a regularization term which is gradually turned to zero. This closely connects to, and explains, what is called overtraining in the neural net literature.