Learning and retrieving spatio-temporal sequences with any static associative neural network

The purpose of the present paper is to report on the design of a general system that is capable of learning and retrieving spatio-temporal sequences using any static associative neural networks (ANNs), including both autoassociative and heteroassociative neural networks. This artificial neural system has three major components: a voting network, a parallel array of ANN's, and delayed feedback lines from the output of the system to the ANN layer. The system has separate primary and pairing input channels. During learning, pairs of sequences of spatial patterns are presented to the primary and the pairing input channels simultaneously. During retrieving, a cue sequence, which is presented to the primary input channel only, and which may have spatial imperfections and temporal gaps, causes the system to output the stored spatio-temporal sequence. As a demonstration of the applicability of the present general system, we present an implementation using a specific neural network, that is, our dynamically generated variant of the counterpropagation network. This system shows computational advantages such as fast and accurate learning and retrieving, and the ability to store a large number of sequences consisting of nonorthogonal spatial patterns.

[1]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[2]  Michael A. Arbib,et al.  Timing and chunking in processing temporal order , 1993, IEEE Trans. Syst. Man Cybern..

[3]  Personnaz,et al.  Storage and retrieval of complex sequences in neural networks. , 1988, Physical review. A, General physics.

[4]  DeLiang Wang,et al.  Anticipation-based temporal pattern generation , 1995, IEEE Trans. Syst. Man Cybern..

[5]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[6]  Lipo Wang,et al.  Oscillatory and chaotic dynamics in neural networks under varying operating conditions , 1996, IEEE Trans. Neural Networks.

[7]  Lipo Wang,et al.  An artificial neural network system for temporal-spatial sequence processing , 1995, Pattern Recognit..

[8]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[9]  Coolen,et al.  Image evolution in Hopfield networks. , 1988, Physical review. A, General physics.

[10]  Lipo Wang,et al.  Artificial neural networks - oscillations, chaos, and sequence processing , 1993 .

[11]  Thomas P. Caudell,et al.  A neural architecture for pattern sequence verification through inferencing , 1993, IEEE Trans. Neural Networks.

[12]  S. Grossberg,et al.  Neural Dynamics of Category Learning and Recognition: Attention, Memory Consolidation, and Amnesia , 1987 .

[13]  Wang,et al.  Chaos, multiplicity, crisis, and synchronicity in higher-order neural networks. , 1991, Physical review. A, Atomic, molecular, and optical physics.

[14]  Michael A. Arbib,et al.  Complex temporal sequence learning based on short-term memory , 1990 .

[15]  D Kleinfeld,et al.  Sequential state generation by model neural networks. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[16]  S Dehaene,et al.  Neural networks that learn temporal sequences by selection. , 1987, Proceedings of the National Academy of Sciences of the United States of America.

[17]  Kanter,et al.  Temporal association in asymmetric neural networks. , 1986, Physical review letters.

[18]  B. A. Minch,et al.  IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing , 1998 .

[19]  Kenji Doya,et al.  Adaptive neural oscillator using continuous-time back-propagation learning , 1989, Neural Networks.

[20]  P. Peretto,et al.  Collective Properties of Neural Networks , 1986 .

[21]  Richard P. Lippmann,et al.  Review of Neural Networks for Speech Recognition , 1989, Neural Computation.

[22]  Albert Nigrin,et al.  Neural networks for pattern recognition , 1993 .

[23]  Steven K. Rogers,et al.  EFFECTIVE NEURAL NETWORK MODELING IN C , 1991 .

[24]  Stephen Grossberg,et al.  Fast-learning VIEWNET architectures for recognizing three-dimensional objects from multiple two-dimensional views , 1995, Neural Networks.

[25]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[26]  Shiino,et al.  Retrieval of spatio-temporal sequence in asynchronous neural network. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[27]  Lizhong Wu,et al.  On the design of connectionist vector quantizers , 1991 .

[28]  Stephen Grossberg,et al.  Working Memory Networks for Learning Temporal Order with Application to Three-Dimensional Visual Object Recognition , 1992, Neural Computation.

[29]  Tzi-Dar Chiueh,et al.  Multivalued associative memories based on recurrent networks , 1993, IEEE Trans. Neural Networks.

[30]  Joachim M. Buhmann,et al.  Noise-driven temporal association in neural networks , 1987 .

[31]  L. Wang,et al.  Oscillations and chaos in neural networks: an exactly solvable model. , 1990, Proceedings of the National Academy of Sciences of the United States of America.

[32]  Ah Chung Tsoi,et al.  Locally recurrent globally feedforward networks: a critical review of architectures , 1994, IEEE Trans. Neural Networks.

[33]  DeLiang Wang,et al.  Incremental learning of complex temporal patterns , 1996, IEEE Trans. Neural Networks.

[34]  John J. Hopfield,et al.  Connected-digit speaker-dependent speech recognition using a neural network with time-delayed connections , 1991, IEEE Trans. Signal Process..

[35]  BART KOSKO,et al.  Bidirectional associative memories , 1988, IEEE Trans. Syst. Man Cybern..

[36]  Stephen Grossberg,et al.  ARTMAP: supervised real-time learning and classification of nonstationary data by a self-organizing neural network , 1991, [1991 Proceedings] IEEE Conference on Neural Networks for Ocean Engineering.

[37]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[38]  J J Hopfield,et al.  Neural computation by concentrating information in time. , 1987, Proceedings of the National Academy of Sciences of the United States of America.

[39]  Lipo Wang.  Suppressing chaos with hysteresis in a higher order neural network , 1996 .