Extended Kalman filter neural network training: experimental results and algorithm improvements

It is well known that the extended Kalman filter (EKF) neural network training algorithm is superior to the standard backpropagation algorithm. However, there are many variations on the EKF implementation that can significantly affect its performance. For example, improper initialization of three parameters cause the algorithm to perform poorly. There are also two advanced methods, decoupling and multistreaming, which need to be properly applied based on the specifics of the problem. This paper presents the results of extensive experimentation in applying the EKF training method for recurrent and static neural networks. The goal is to demonstrate how different variations on its implementation effect performance and to find methods to optimize performance. The paper examines the effects of decoupling, multistreaming, and initial values of constants used by the algorithm. Three new ideas are suggested that can lead to improved performance. These ideas are: initializing parameters to values outside the range previously suggested, a new decoupling strategy, and reducing the update rate of the error covariance matrix for faster training.

[1]  F. Heimes,et al.  Traditional and evolved dynamic neural networks for aircraft simulation , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[2]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[3]  G. V. Puskorius,et al.  Training controllers for robustness: multi-stream DEKF , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[4]  Sharad Singhal,et al.  Training Multilayer Perceptrons with the Extende Kalman Algorithm , 1988, NIPS.

[5]  G. V. Puskorius,et al.  Multi-stream extended Kalman filter training for static and dynamic neural networks , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[6]  Lee A. Feldkamp,et al.  Decoupled extended Kalman filter training of feedforward layered networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.