Relating Real-Time Backpropagation and Backpropagation-Through-Time: An Application of Flow Graph Interreciprocity

We show that signal flow graph theory provides a simple way to relate two popular algorithms used for adapting dynamic neural networks, real-time backpropagation and backpropagation-through-time. Starting with the flow graph for real-time backpropagation, we use a simple transposition to produce a second graph. The new graph is shown to be interreciprocal with the original and to correspond to the backpropagation-through-time algorithm. Interreciprocity provides a theoretical argument to verify that both flow graphs implement the same overall weight update.

[1]  B. Widrow,et al.  Neural networks for self-learning control systems , 1990, IEEE Control Systems Magazine.

[2]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[3]  E. S. Plumer.  Time-optimal terminal control using neural networks , 1993, IEEE International Conference on Neural Networks.

[4]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[5]  S. Ramo,et al.  Fields and Waves in Communication Electronics , 1966 .

[6]  P J Webros.  BACKPROPAGATION THROUGH TIME: WHAT IT DOES AND HOW TO DO IT , 1990 .

[7]  R. Spence,et al.  Tellegen's theorem and electrical networks , 1970 .

[8]  S. Piche,et al.  First-Order Gradient Descent Training of Adaptive Discrete-Time Dynamic Networks , 1991 .

[9]  Geoffrey E. Hinton,et al.  Proceedings of the 1988 Connectionist Models Summer School , 1989 .

[10]  Derrick H. Nguyen,et al.  Neural networks for self-learning control systems , 1990 .

[11]  Yann LeCun,et al.  A theoretical framework for back-propagation , 1988 .

[12]  Franklin A. Graybill,et al.  Introduction to The theory , 1974 .

[13]  S. W. Piche,et al.  Steepest descent algorithms for neural network controllers and filters , 1994, IEEE Trans. Neural Networks.

[14]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[15]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[16]  A. Gemant,et al.  Ion formation and inhibition in hydrocarbons , 1956 .

[17]  Yann Le Cun,et al.  A Theoretical Framework for Back-Propagation , 1988 .

[18]  Pineda,et al.  Generalization of back-propagation to recurrent neural networks. , 1987, Physical review letters.

[19]  K S Narendra,et al.  IDENTIFICATION AND CONTROL OF DYNAMIC SYSTEMS USING NEURAL NETWORKS , 1990 .

[20]  Arthur E. Bryson,et al.  Applied Optimal Control , 1969 .

[21]  J. Bordewijk.  Inter-reciprocity applied to electrical networks , 1957 .

[22]  Thomas Kailath,et al.  Linear Systems , 1980 .