Recurrent network training with the decoupled-extended-Kalman-filter algorithm

In this paper we describe the extension of our decoupled extended Kalman filter (DEKF) training algorithm to networks with internal recurrent (or feedback) connections; we call the resulting algorithm dynamic DEKF (or DDEKF for short). Analysis of DDEKF's computational complexity and empirical evidence suggest significant computational and performance advantages in comparison to training algorithms based exclusively upon gradient descent. We demonstrate DDEKF's effectiveness by training networks with recurrent connections for four different classes of problems. First, DDEKF is used to train a recurrent network that produces as its output a delayed copy of its input. Second, recurrent networks are trained by DDEKF to recognize sequences of events with arbitrarily long time delays between the events. Third, DDEKF is applied to the training of identification networks to act as models of the input-output behavior for nonlinear dynamical systems. We conclude the paper with a brief discussion of the extension of DDEKF to the training of neural controllers with internal feedback connections.

[1]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[2]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[3]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[4]  Ronald J. Williams,et al.  Adaptive state representation and estimation using recurrent connectionist networks , 1990 .

[5]  Alexander G. Parlos,et al.  Nonlinear dynamic system identification using artificial neural networks (ANNs) , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[6]  Lee A. Feldkamp,et al.  Decoupled extended Kalman filter training of feedforward layered networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[7]  Donald F. Specht,et al.  A general regression neural network , 1991, IEEE Trans. Neural Networks.

[8]  C. L. Giles,et al.  Second-order recurrent neural networks for grammatical inference , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.