Relationships Between the A Priori and A Posteriori Errors in Nonlinear Adaptive Neural Filters

The lower bounds for the a posteriori prediction error of a nonlinear predictor realized as a neural network are provided. These are obtained for a priori adaptation and a posteriori error networks with sigmoid nonlinearities trained by gradient-descent learning algorithms. A contractivity condition is imposed on a nonlinear activation function of a neuron so that the a posteriori prediction error is smaller in magnitude than the corresponding a priori one. Furthermore, an upper bound is imposed on the learning rate so that the approach is feasible. The analysis is undertaken for both feedforward and recurrent nonlinear predictors realized as neural networks.

[1]  C. Johnson,et al.  Theory and design of adaptive filters , 1987 .

[2]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[3]  Danilo P. Mandic,et al.  Global asymptotic convergence of nonlinear relaxation equations realised through a recurrent perceptron , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[4]  D. Luenberger.  Optimization by Vector Space Methods , 1968 .

[5]  Philip E. Gill,et al.  Practical optimization , 1981 .

[6]  S. Singh.  Nonlinear Functional Analysis and Its Applications , 1986 .

[7]  Danilo P. Mandic,et al.  Relating the Slope of the Activation Function and the Learning Rate Within a Recurrent Neural Network , 1999, Neural Computation.

[8]  S. Douglas,et al.  A posteriori updates for adaptive filters , 1997, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136).

[9]  Danilo P. Mandic,et al.  A posteriori real-time recurrent learning schemes for a recurrent neural network based nonlinear predictor , 1998 .

[10]  Kumpati S. Narendra,et al.  Gradient methods for the optimization of dynamical systems containing neural networks , 1991, IEEE Trans. Neural Networks.

[11]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[12]  Emile Fiesler,et al.  The Interchangeability of Learning Rate and Gain in Backpropagation Neural Networks , 1996, Neural Computation.

[13]  Simon Haykin,et al.  Adaptive filter theory (2nd ed.) , 1991 .

[14]  Lennart Ljung,et al.  Theory and Practice of Recursive Identification , 1983 .

[15]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.