Backpropagation Algorithms for a Broad Class of Dynamic Networks

This paper introduces a general framework for describing dynamic neural networks-the layered digital dynamic network (LDDN). This framework allows the development of two general algorithms for computing the gradients and Jacobians for these dynamic networks: backpropagation-through-time (BPTT) and real-time recurrent learning (RTRL). The structure of the LDDN framework enables an efficient implementation of both algorithms for arbitrary dynamic networks. This paper demonstrates that the BPTT algorithm is more efficient for gradient calculations, but the RTRL algorithm is more efficient for Jacobian calculations

[1]  M. Hagan,et al.  TRAINING RECURRENT NETWORKS FOR FILTERING AND CONTROL , 1999 .

[2]  Eric A. Wan,et al.  Temporal backpropagation for FIR neural networks , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[3]  Jacob Barhen,et al.  Learning a trajectory using adjoint functions and teacher forcing , 1992, Neural Networks.

[4]  Danil V. Prokhorov,et al.  Phased backpropagation: a hybrid of BPTT and temporal BP , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[5]  Orlando De Jesus,et al.  Training general dynamic neural networks , 2002 .

[6]  Stefan C. Kremer,et al.  Spatiotemporal Connectionist Networks: A Taxonomy and Review , 2001, Neural Computation.

[7]  K S Narendra,et al.  IDENTIFICATION AND CONTROL OF DYNAMIC SYSTEMS USING NEURAL NETWORKS , 1990 .

[8]  Lee A. Feldkamp,et al.  Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks , 1994, IEEE Trans. Neural Networks.

[9]  Madan M. Gupta,et al.  Static and Dynamic Neural Networks: From Fundamentals to Advanced Theory , 2003 .

[10]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[11]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[12]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[13]  Ronald J. Williams,et al.  Gradient-based learning algorithms for recurrent networks and their computational complexity , 1995 .

[14]  Ah Chung Tsoi,et al.  Locally recurrent globally feedforward networks: a critical review of architectures , 1994, IEEE Trans. Neural Networks.

[15]  Eric A. Wan,et al.  Diagrammatic Methods for Deriving and Relating Temporal Neural Network Algorithms , 1997, Summer School on Neural Networks.

[16]  Alexander G. Parlos,et al.  Nonlinear dynamic system identification using artificial neural networks (ANNs) , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[17]  Peter Tiño,et al.  Markovian architectural bias of recurrent neural networks , 2004, IEEE Transactions on Neural Networks.

[18]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[19]  S. Sra,et al.  Matrix Differential Calculus , 2005 .

[20]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[21]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[22]  Amir F. Atiya,et al.  New results on recurrent network training: unifying the algorithms and accelerating convergence , 2000, IEEE Trans. Neural Networks Learn. Syst..

[23]  Tet Hin Yeap,et al.  Decision feedback recurrent neural equalization with fast convergence rate , 2005, IEEE Transactions on Neural Networks.

[24]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[25]  Paolo Campolucci,et al.  Signal-flow-graph derivation of on-line gradient learning algorithms , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[26]  M. W. Pedersen,et al.  Training recurrent networks , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[27]  Nikita Barabanov,et al.  Stability analysis of discrete-time recurrent neural networks , 2002, IEEE Trans. Neural Networks.

[28]  Martin T. Hagan,et al.  Neural network design , 1995 .

[29]  Paul J. Werbos,et al.  The roots of backpropagation , 1994 .

[30]  Jürgen Schmidhuber,et al.  A Fixed Size Storage O(n3) Time Complexity Learning Algorithm for Fully Recurrent Continually Running Networks , 1992, Neural Computation.

[31]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[32]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction , 2001 .

[33]  Jacob Barhen,et al.  Adjoint-Functions and Temporal Learning Algorithms in Neural Networks , 1990, NIPS.

[34]  M.T. Hagan,et al.  Backpropagation through time for a general class of recurrent network , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[35]  G. V. Puskorius,et al.  A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification , 1998, Proc. IEEE.