Classification of temporal sequences via prediction using the simple recurrent neural network

Abstract An approach to classify temporal sequences using the simple recurrent neural network (SRNN) is developed in this paper. A classification problem is formulated as a component prediction problem and two training methods are described to train a single SRNN to predict the components of temporal sequences belonging to multiple classes. Issues related to the selection of the dimension of the context vector and the influence of the context vector on classification are identified and investigated. The use of a different initial context vector for each class is proposed as a means to improve classification and a classification rule which incorporates the different initial context vectors is formulated. A systematic method in which the SRNN is trained with noisy exemplars is developed to enhance the classification performance of the network. A 4-class localized object classification problem is selected to demonstrate that (a) a single SRNN can be trained to classify real multi-class sequences via component prediction, (b) the classification accuracy can be improved by using a distinguishing initial context vector for each class, and (c) the classification accuracy of the SRNN can be improved significantly by using the distinguishing initial context vector in conjunction with the systematic re-training method. It is concluded that, through the approach developed in this paper, the SRNN can robustly classify temporal sequences which may have an unequal number of components.

[1]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[2]  Terrence J. Sejnowski,et al.  NETtalk: a parallel network that learns to read aloud , 1988 .

[3]  Yuan-Fu Liao,et al.  Speech recognition with hierarchical recurrent neural networks , 1995, Pattern Recognit..

[4]  Barak A. Pearlmutter.  Learning State Space Trajectories in Recurrent Neural Networks , 1989, Neural Computation.

[5]  Olivier D. Faugeras,et al.  HYPER: A New Approach for the Recognition and Positioning of Two-Dimensional Objects , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Bernhard R. Kämmerer,et al.  Experiments for isolated-word recognition with single- and two-layer perceptrons , 1990, Neural Networks.

[7]  Fernando J. Pineda,et al.  Generalization of Back propagation to Recurrent and Higher Order Neural Networks , 1987, NIPS.

[8]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[9]  Mohammad R. Sayeh,et al.  A neural network approach to robust shape classification , 1990, Pattern Recognit..

[10]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[11]  C. Lee Giles,et al.  Pruning recurrent neural networks for improved generalization performance , 1994, IEEE Trans. Neural Networks.

[12]  James L. McClelland,et al.  Finite State Automata and Simple Recurrent Networks , 1989, Neural Computation.

[13]  Jeffrey L. Elman,et al.  Distributed Representations, Simple Recurrent Networks, and Grammatical Structure , 1991, Mach. Learn..

[14]  Owen Robert Mitchell,et al.  Partial Shape Recognition Using Dynamic Programming , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Robert F. Port,et al.  Representation and Recognition of Temporal Patterns , 1990 .

[16]  Antonette M. Logar,et al.  A comparison of recurrent neural network learning algorithms , 1993, IEEE International Conference on Neural Networks.

[17]  Alexander H. Waibel,et al.  Modular Construction of Time-Delay Neural Networks for Speech Recognition , 1989, Neural Computation.

[18]  Ramesh C. Jain,et al.  Recognizing partially visible objects using feature indexed hypotheses , 1986, IEEE J. Robotics Autom..

[19]  Lalit Gupta,et al.  Three-layer perceptron based classifiers for the partial shape classification problem , 1994, Pattern Recognit..

[20]  Rangasami L. Kashyap,et al.  Using Polygons to Recognize and Locate Partially Occluded Objects , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  R. Eckmiller,et al.  Neural Computers , 1989, Springer Study Edition.

[22]  Lalit Gupta,et al.  Investigating the prediction capabilities of the simple recurrent neural network on real temporal sequences , 2000, Pattern Recognit..

[23]  Lalit Gupta,et al.  Robust automatic target recognition using a localized boundary representation , 1995, Pattern Recognit..

[24]  Richard A. Volz,et al.  Recognizing Partially Occluded Parts , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Lalit Gupta,et al.  Robust partial shape classification using invariant breakpoints and dynamic alignment , 1990, Pattern Recognit..

[26]  Vijay Karamcheti,et al.  Sequence learning with recurrent networks: analysis of internal representations , 1992, Defense, Security, and Sensing.

[27]  Mandyam D. Srinath,et al.  Partial Shape Classification Using Contour Matching in Distance Transformation , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  C. Lee Giles,et al.  An analysis of noise in recurrent neural networks: convergence and generalization , 1996, IEEE Trans. Neural Networks.

[29]  Alois Goller,et al.  Parallel and Distributed Processing , 1998, Lecture Notes in Computer Science.

[30]  Alan F. Murray,et al.  Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training , 1994, IEEE Trans. Neural Networks.

[31]  L. Gupta,et al.  Non-linear alignment of neural net outputs for partial shape classification , 1991, Pattern Recognit..

[32]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[33]  Paul W. Munro,et al.  Nets with Unreliable Hidden Nodes Learn Error-Correcting Codes , 1992, NIPS.

[34]  D Zipser,et al.  Learning the hidden structure of speech. , 1988, The Journal of the Acoustical Society of America.

[35]  DeLiang Wang,et al.  On Temporal Generalization of Simple Recurrent Networks , 1996, Neural Networks.

[36]  Geoffrey E. Hinton,et al.  A time-delay neural network architecture for isolated word recognition , 1990, Neural Networks.

[37]  L. B. Lmeida.  Backpropagation in perceptrons with feedback , 1988 .