Dual extended Kalman filtering in recurrent neural networks

In the classical deterministic Elman model, the estimation of parameters must be very accurate. Otherwise, the system performance is very poor. To improve the system performance, we can use a Kalman filtering algorithm to guide the operation of a trained recurrent neural network (RNN). In this case, during training, we need to estimate the state of hidden layer, as well as the weights of the RNN. This paper discusses how to use the dual extended Kalman filtering (DEKF) for this dual estimation and how to use our proposing DEKF for removing some unimportant weights from a trained RNN. In our approach, one Kalman algorithm is used for estimating the state of the hidden layer, and one recursive least square (RLS) algorithm is used for estimating the weights. After training, we use the error covariance matrix of the RLS algorithm to remove unimportant weights. Simulation showed that our approach is an effective joint-learning-pruning method for RNNs under the online operation.

[1]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[2]  Nazif Tepedelenlioglu,et al.  A fast new algorithm for training feedforward neural networks , 1992, IEEE Trans. Signal Process..

[3]  Jürgen Schmidhuber,et al.  A Fixed Size Storage O(n3) Time Complexity Learning Algorithm for Fully Recurrent Continually Running Networks , 1992, Neural Computation.

[4]  Lee A. Feldkamp,et al.  Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks , 1994, IEEE Trans. Neural Networks.

[5]  Barak A. Pearlmutter.  Fast Exact Multiplication by the Hessian , 1994, Neural Computation.

[6]  Kwok-Wo Wong,et al.  A pruning method for the recursive least squared algorithm , 2001, Neural Networks.

[7]  David Zipser,et al.  Subgrouping Reduces Complexity and Speeds Up Learning in Recurrent Networks , 1989, NIPS.

[8]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[9]  Francesco Palmieri,et al.  Optimal filtering algorithms for fast learning in feedforward neural networks , 1992, Neural Networks.

[10]  J. Sum,et al.  Extended Kalman FilterBased Pruning Method for Recurrent Neural Networks , 1998, Neural Computation.

[11]  David E. Rumelhart,et al.  Generalization by Weight-Elimination with Application to Forecasting , 1990, NIPS.

[12]  Lars Kai Hansen,et al.  On design and evaluation of tapped-delay neural network architectures , 1993, IEEE International Conference on Neural Networks.

[13]  Hideaki Sakai,et al.  A real-time learning algorithm for a multilayered neural network based on the extended Kalman filter , 1992, IEEE Trans. Signal Process..

[14]  Jan Larsen,et al.  DESIGN OF NEURAL NETWORK FILTERS , 1996 .

[15]  Ah Chung Tsoi,et al.  Recurrent neural networks: A constructive algorithm, and its properties , 1997, Neurocomputing.

[16]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[17]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[18]  Eric A. Wan,et al.  Neural dual extended Kalman filtering: applications in speech enhancement and monaural blind signal separation , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[19]  Lai-Wan Chan,et al.  On-line training and pruning for recursive least square algorithms , 1996 .