DESIGN OF NEURAL NETWORK FILTERS

The subject of this Ph.D. Thesis is design of neural network filters. Neural network filters may be viewed as an extension of classical linear adaptive filters to deal with nonlinear modeling tasks. We focus on neural network architectures for implementation of the non-recursive, nonlinear adaptive model with additive error. The objective is to clarify a number of phases involved in the design of neural network filter architectures in connection with “black box” modeling tasks such as system identification, inverse modeling and timeseries prediction. The major contributions comprises: • The development of an architecture taxonomy based on formulating a canonical filter representation. The substantial part of the taxonomy is the distinction between global and local models. The taxonomy leads to the classification of a number of existing neural network architectures and, in addition, suggests the potential development of novel structures. Various architectures are reviewed and interpreted. Especially we attach importance to interpretations of the multi-layer perceptron neural network. • Formulation of a generic nonlinear filter architecture which consists of a combination of the canonical filter and a preprocessing unit. The architecture may be viewed as a heterogeneous three-layer neural network. A number of preprocessing methods are suggested with reference to bypassing the “curse of dimensionality” without reducing the performance significantly. • Discussion of various algorithms for estimating characteristic model weights (parameters). We suggest efficient implementations of standard first and second order optimization algorithms for layered architectures. In addition, in order to speedup convergence a weight initialization algorithm for the 2-layer perceptron neural networks is developed. • Clarification and discussion of fundamental limitations in the search for optimal network architectures based upon a decomposition of the average generalization error, called the model error decomposition. This includes a discussion of employing regularization. • The development and discussion of a novel generalization error estimator, GEN , which is valid for incomplete, nonlinear models. The ability to deal with incomplete models is particularly important when performing “black box” modeling. The models are assumed to be estimated by minimizing the least squares cost function with a regularization term. The estimator is based on a statistical framework and may be viewed as an extension of Akaike’s classical FPE -estimator and Moody’s GPE -estimator.