Recurrent Neural Network Architectures: An Overview

In this paper, we have first considered a number of popular recurrent neural network architectures. Then, two subclasses of general recurrent neural network architectures are introduced. It is shown that all these popular recurrent neural network architectures can be grouped under either of these two subclasses of general recurrent neural network architectures. It is also inferred that these two subclasses of recurrent neural network architectures are distinct, in that it is not possible to transform from one form to the other. Two recently introduced recurrent neural network architectures specifically designed for special purposes, viz., for overcoming long term temporal dependency, and for data structure classifications are also considered.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  Stephen A. Billings,et al.  Non-linear system identification using neural networks , 1990 .

[3]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[4]  Giovanni Soda,et al.  Local Feedback Multilayered Networks , 1992, Neural Computation.

[5]  Eduardo D. Sontag,et al.  For neural networks, function determines form , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[6]  Alessandro Sperduti,et al.  Labelling Recursive Auto-associative Memory , 1994, Connect. Sci..

[7]  Alessandro Sperduti,et al.  Supervised neural networks for the classification of structures , 1997, IEEE Trans. Neural Networks.

[8]  José Carlos Príncipe,et al.  The gamma-filter-a new class of adaptive IIR filters with restricted feedback , 1993, IEEE Trans. Signal Process..

[9]  Eric A. Wan,et al.  Temporal backpropagation for FIR neural networks , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[10]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[11]  Ah Chung Tsoi,et al.  FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling , 1991, Neural Computation.

[12]  Thomas Kailath,et al.  Linear Systems , 1980 .

[13]  Fernando J. Pineda,et al.  Dynamics and architecture for neural computation , 1988, J. Complex..

[14]  Dirk Grunwald,et al.  Evidence-based static branch prediction using machine learning , 1997, TOPL.

[15]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[16]  Hava T. Siegelmann,et al.  Computational capabilities of recurrent NARX neural networks , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[17]  Michael I. Jordan.  Supervised learning and systems with excess degrees of freedom , 1988 .

[18]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[19]  Ah Chung Tsoi,et al.  Discrete time recurrent neural network architectures: A unifying review , 1997, Neurocomputing.

[20]  Ah Chung Tsoi,et al.  Locally recurrent globally feedforward networks: a critical review of architectures , 1994, IEEE Trans. Neural Networks.

[21]  Eduardo D. Sontag,et al.  Neural Networks for Control , 1993 .

[22]  Ah Chung Tsoi,et al.  Gradient Based Learning Methods , 1997, Summer School on Neural Networks.

[23]  Pierre Baldi,et al.  Hybrid Modeling, HMM/NN Architectures, and Protein Applications , 1996, Neural Computation.

[24]  Ah Chung Tsoi,et al.  Universal Approximation Using Feedforward Neural Networks: A Survey of Some Existing Methods, and Some New Results , 1998, Neural Networks.

[25]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[26]  Pierre Roussel-Ragot,et al.  Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms , 1993, Neural Computation.

[27]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[28]  Ah Chung Tsoi,et al.  Application of Neural Network Methodology to the Modelling of the Yield Strength in a Steel Rolling Plate Mill , 1991, NIPS.

[29]  Albert Y. Zomaya,et al.  Neuro-Adaptive Process Control: A Practical Approach , 1996 .

[30]  A. Lapedes,et al.  Nonlinear signal processing using neural networks: Prediction and system modelling , 1987 .

[31]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .