Discrete time recurrent neural network architectures: A unifying review

Abstract In this paper, after giving definitions for a set of commonly used terms in recurrent neural networks (RNNs), all possible RNN architectures based on these definitions are enumerated, and described. Then, most existing RNN architectures are categorized under these headings. Four general neural network architectures, in increasing degree of complexity, are introduced. It is shown that all the existing RNN architectures can be considered as special cases of the general RNN architectures. Furthermore, it is shown how these existing architectures can be transformed to the general RNN architectures. Some open issues concerning RNN architectures are discussed.

[1]  A. Lapedes,et al.  Nonlinear signal processing using neural networks: Prediction and system modelling , 1987 .

[2]  Alberto Del Bimbo,et al.  Block-structured recurrent neural networks , 1995, Neural Networks.

[3]  Les E. Atlas,et al.  Recurrent neural networks and robust time series prediction , 1994, IEEE Trans. Neural Networks.

[4]  Fernando J. Pineda,et al.  Dynamics and architecture for neural computation , 1988, J. Complex..

[5]  P. S. Sastry,et al.  Memory neuron networks for identification and control of dynamical systems , 1994, IEEE Trans. Neural Networks.

[6]  Ronald J. Williams,et al.  Experimental Analysis of the Real-time Recurrent Learning Algorithm , 1989 .

[7]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[8]  Bart W. Stuck,et al.  A Computer and Communication Network Performance Analysis Primer (Prentice Hall, Englewood Cliffs, NJ, 1985; revised, 1987) , 1987, Int. CMG Conference.

[9]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[10]  José Carlos Príncipe,et al.  The gamma model--A new neural model for temporal processing , 1992, Neural Networks.

[11]  Eric A. Wan,et al.  Temporal backpropagation for FIR neural networks , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[12]  Ah Chung Tsoi,et al.  A Comparison of Discrete-Time Operator Models and for Nonlinear System Identification , 1994, NIPS.

[13]  U. R. Prasad,et al.  Back propagation through adjoints for the identification of nonlinear dynamic systems using recurrent neural models , 1994, IEEE Trans. Neural Networks.

[14]  P. Gallman,et al.  An iterative method for the identification of nonlinear systems using a Uryson model , 1975 .

[15]  Giovanni Soda,et al.  Local Feedback Multilayered Networks , 1992, Neural Computation.

[16]  Ah Chung Tsoi,et al.  FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling , 1991, Neural Computation.

[17]  D. Friedman,et al.  On approximating an FIR filter using discrete orthonormal exponentials , 1981 .

[18]  S. W. Piche,et al.  Steepest descent algorithms for neural network controllers and filters , 1994, IEEE Trans. Neural Networks.

[19]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[20]  Yoshua Bengio,et al.  Learning the dynamic nature of speech with back-propagation for sequences , 1992, Pattern Recognit. Lett..

[21]  C. Lee Giles,et al.  An experimental comparison of recurrent neural networks , 1994, NIPS.

[22]  David J. N. Limebeer,et al.  Linear Robust Control , 1994 .

[23]  David Zipser,et al.  A Subgrouping Strategy that Reduces Complexity and Speeds Up Learning in Recurrent Networks , 1989, Neural Computation.

[24]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[25]  Martin Brown,et al.  Neurofuzzy adaptive modelling and control , 1994 .

[26]  David L. Elliott,et al.  A Better Activation Function for Artificial Neural Networks , 1993 .

[27]  Albert Y. Zomaya,et al.  Adaptive Model-Based Control using Neural Networks , 1993 .

[28]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[29]  Yung C. Shin,et al.  Radial basis function neural network for approximation and estimation of nonlinear stochastic dynamic systems , 1994, IEEE Trans. Neural Networks.

[30]  Amir F. Atiya,et al.  Application of the recurrent multilayer perceptron in modeling complex process dynamics , 1994, IEEE Trans. Neural Networks.

[31]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[32]  Hava T. Siegelmann,et al.  Computational capabilities of recurrent NARX neural networks , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[33]  Ah Chung Tsoi,et al.  An Adaptive Lattice Architecture for Dynamic Multilayer Perceptrons , 1992, Neural Computation.

[34]  Bart Kosko,et al.  Neural networks for signal processing , 1992 .

[35]  Malur K. Sundareshan,et al.  Identification and decentralized adaptive control using dynamical neural networks with application to robotic manipulators , 1993, IEEE Trans. Neural Networks.

[36]  Michael I. Jordan.  Supervised learning and systems with excess degrees of freedom , 1988 .

[37]  P. Gallman.  A comparison of two Hammerstein model identification algorithms , 1976 .

[38]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[39]  Darrell Williamson,et al.  Delay replacement in direct form structures , 1988, IEEE Trans. Acoust. Speech Signal Process..

[40]  Ah Chung Tsoi,et al.  Locally recurrent globally feedforward networks: a critical review of architectures , 1994, IEEE Trans. Neural Networks.

[41]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[42]  Sheng Chen.  Nonlinear time series modelling and prediction using Gaussian RBF networks with enhanced clustering and RLS learning , 1995 .

[43]  Thomas Kailath,et al.  Linear Systems , 1980 .

[44]  Stephen A. Billings,et al.  Non-linear system identification using neural networks , 1990 .

[45]  Ah Chung Tsoi,et al.  Representational Capabilities of Multilayer Feedforward Networks with Time-Delay Synapses , 1992, 1992 American Control Conference.

[46]  Pierre Roussel-Ragot,et al.  Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms , 1993, Neural Computation.

[47]  Luís B. Almeida,et al.  A learning rule for asynchronous perceptrons with feedback in a combinatorial environment , 1990 .

[48]  K. P. Unnikrishnan,et al.  Nonlinear prediction of speech signals using memory neuron networks , 1991, Neural Networks for Signal Processing Proceedings of the 1991 IEEE Workshop.

[49]  Fernando J. Pineda,et al.  Recurrent Backpropagation and the Dynamical Approach to Adaptive Neural Computation , 1989, Neural Computation.

[50]  W. M. Carey,et al.  Digital spectral analysis: with applications , 1986 .

[51]  Manoel Fernando Tenorio,et al.  A neural computation model with short-term memory , 1995, IEEE Trans. Neural Networks.

[52]  Kwang Y. Lee,et al.  Diagonal recurrent neural networks for dynamic systems control , 1995, IEEE Trans. Neural Networks.

[53]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[54]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[55]  Graham C. Goodwin,et al.  Adaptive filtering prediction and control , 1984 .

[56]  Kumpati S. Narendra,et al.  Identification Using Feedforward Networks , 1995, Neural Computation.

[57]  Bernie Mulgrew,et al.  IEEE Workshop on Neural Networks for Signal Processing , 1995 .

[58]  José Carlos Príncipe,et al.  A Theory for Neural Networks with Time Delays , 1990, NIPS.

[59]  M. Gori,et al.  BPS: a learning algorithm for capturing the dynamic nature of speech , 1989, International 1989 Joint Conference on Neural Networks.

[60]  A. C. Tsoi,et al.  Nonlinear system identification using multilayer perceptrons with locally recurrent synaptic structure , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[61]  Heinz Unbehauen,et al.  Structure identification of nonlinear dynamic systems - A survey on input/output approaches , 1990, Autom..

[62]  J.C. Principe,et al.  Adaline with adaptive recursive memory , 1991, Neural Networks for Signal Processing Proceedings of the 1991 IEEE Workshop.