A Taxonomy for Spatiotemporal Connectionist Networks Revisited: The Unsupervised Case

Spatiotemporal connectionist networks (STCNs) comprise an important class of neural models that can deal with patterns distributed in both time and space. In this article, we widen the application domain of the taxonomy for supervised STCNs recently proposed by Kremer (2001) to the unsupervised case. This is possible through a reinterpretation of the state vector as a vector of latent (hidden) variables, as proposed by Meinicke (2000). The goal of this generalized taxonomy is then to provide a nonlinear generative framework for describing unsupervised spatiotemporal networks, making it easier to compare and contrast their representational and operational characteristics. Computational properties, representational issues, and learning are also discussed, and a number of references to the relevant source publications are provided. It is argued that the proposed approach is simple and more powerful than the previous attempts from a descriptive and predictive viewpoint. We also discuss the relation of this taxonomy with automata theory and state-space modeling and suggest directions for further work.

[1]  Aluizio F. R. Araújo,et al.  Context in temporal sequence processing: a self-organizing approach and its application to robotics , 2002, IEEE Trans. Neural Networks.

[2]  Narendra Ahuja,et al.  A topological and temporal correlator network for spatiotemporal pattern learning, recognition, and recall , 1999, IEEE Trans. Neural Networks.

[3]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[4]  Peter Dayan,et al.  Factor Analysis Using Delta-Rule Wake-Sleep Learning , 1997, Neural Computation.

[5]  Erkki Oja,et al.  Neural and statistical classifiers-taxonomy and two case studies , 1997, IEEE Trans. Neural Networks.

[6]  Sun-Yuan Kung,et al.  Gradient Adaptive Algorithms for Contrast-Based Blind Deconvolution , 2000, J. VLSI Signal Process..

[7]  Lei Xu,et al.  BYY harmony learning, independent state space, and generalized APT financial analyses , 2001, IEEE Trans. Neural Networks.

[8]  Joachim M. Buhmann,et al.  Noise-driven temporal association in neural networks , 1987 .

[9]  T. Sejnowski,et al.  Storing covariance with nonlinearly interacting neurons , 1977, Journal of mathematical biology.

[10]  Otávio Augusto S. Carpinteiro.  A hierarchical self-organizing map model for sequence recognition , 1998 .

[11]  KEIICHI HORIO,et al.  Feedback Self-Organizing Map and its Application to Spatio-Temporal Pattern Classification , 2001, Int. J. Comput. Intell. Appl..

[12]  T. W. Anderson.  Estimating Linear Statistical Relationships , 1984 .

[13]  Erkki Oja,et al.  Unsupervised learning in neural computation , 2002, Theor. Comput. Sci..

[14]  Jürgen Schmidhuber,et al.  Learning Complex, Extended Sequences Using the Principle of History Compression , 1992, Neural Computation.

[15]  Gen Matsumoto,et al.  Temporal learning rule and dynamic neural network model , 2000, Appl. Math. Comput..

[16]  R. Kempter,et al.  Hebbian learning and spiking neurons , 1999 .

[17]  최승진.  Blind signal deconvolution by spatio-temporal decorrelation and demixing , 1997 .

[18]  Gerald Sommer,et al.  Pattern Recognition by Self-Organizing Neural Networks , 1994 .

[19]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[20]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[21]  Teuvo Kohonen,et al.  Emergence of invariant-feature detectors in the adaptive-subspace self-organizing map , 1996, Biological Cybernetics.

[22]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[23]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[24]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[25]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.

[26]  P. Földiák,et al.  Forming sparse representations by local anti-Hebbian learning , 1990, Biological Cybernetics.

[27]  Allen M. Waxman,et al.  Learning Aspect Graph Representations from View Sequences , 1989, NIPS.

[28]  José Carlos Príncipe,et al.  Competitive principal component analysis for locally stationary time series , 1998, IEEE Trans. Signal Process..

[29]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[30]  A. Klopf.  A neuronal model of classical conditioning , 1988 .

[31]  Risto Miikkulainen,et al.  Modeling the self-organization of directional selectivity in the primary visual cortex , 1999 .

[32]  Robin Sibson,et al.  What is projection pursuit , 1987 .

[33]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[34]  Geoffrey E. Hinton,et al.  Unsupervised learning : foundations of neural computation , 1999 .

[35]  K. L. Adair,et al.  Classification of behavior using unsupervised temporal neural networks , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[36]  Michael Schmitt,et al.  Self-organization of spiking neurons using action potential timing , 1998, IEEE Trans. Neural Networks.

[37]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1990 .

[38]  James V. Stone,et al.  A learning rule for extracting spatio-temporal invariances , 1995 .

[40]  Jörg A. Walter,et al.  Nonlinear prediction with self-organizing maps , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[41]  Roland Baddeley,et al.  Optimal, Unsupervised Learning in Invariant Object Recognition , 1997, Neural Computation.

[42]  Teuvo Kohonen.  THE HYPERMAP ARCHITECTURE , 1991 .

[43]  Guy M. Wallis,et al.  Using Spatio-temporal Correlations to Learn Invariant Object Recognition , 1996, Neural Networks.

[44]  A. E. Maxwell,et al.  Factor Analysis as a Statistical Method. , 1964 .

[45]  Michael C. Mozer,et al.  Beyond Maximum Likelihood and Density Estimation: A Sample-Based Criterion for Unsupervised Learning of Complex Models , 2000, NIPS.

[46]  Stephen Grossberg,et al.  Competitive Learning: From Interactive Activation to Adaptive Resonance , 1987, Cogn. Sci..

[47]  Georg Dorffner,et al.  Neural Networks for Time Series Processing , 1996 .

[48]  José Carlos Príncipe,et al.  Spatio-temporal self-organizing feature maps , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[49]  David Zipser,et al.  Feature Discovery by Competive Learning , 1986, Cogn. Sci..

[50]  Terrence J. Sejnowski,et al.  Unsupervised Learning , 2018, Encyclopedia of GIS.

[51]  Refractor.  Vision , 2000, The Lancet.

[52]  Henrik Jacobsson,et al.  Rule Extraction from Recurrent Neural Networks: ATaxonomy and Review , 2005, Neural Computation.

[53]  Teuvo Kohonen,et al.  Self-Organizing Maps, Second Edition , 1997, Springer Series in Information Sciences.

[54]  Jukka Heikkonen,et al.  Time Series Predicition using Recurrent SOM with Local Linear Models , 1997 .

[55]  Michael Schmitt,et al.  Learning Temporally Encoded Patterns in Networks of Spiking Neurons , 2004, Neural Processing Letters.

[56]  R. C. Williamson,et al.  Regularized principal manifolds , 2001 .

[57]  Jürgen Schmidhuber,et al.  Applying LSTM to Time Series Predictable through Time-Window Approaches , 2000, ICANN.

[58]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[59]  Aluizio F. R. Araújo,et al.  A Self-Organizing Context-Based Approach to the Tracking of Multiple Robot Trajectories , 2002, Applied Intelligence.

[60]  Vijaykumar Gullapalli,et al.  A stochastic reinforcement learning algorithm for learning real-valued functions , 1990, Neural Networks.

[61]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[62]  S. Grossberg.  Some Networks That Can Learn, Remember, and Reproduce any Number of Complicated Space-Time Patterns, I , 1969 .

[63]  Terrence J. Sejnowski,et al.  Unsupervised Discrimination of Clustered Data via Optimization of Binary Information Gain , 1992, NIPS.

[64]  Risto Miikkulainen,et al.  SARDSRN: A Neural Network Shift-Reduce Parser , 1999, IJCAI.

[65]  R. Zemel,et al.  Learning sparse multiple cause models , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[66]  Peter Földiák,et al.  Learning Invariance from Transformation Sequences , 1991, Neural Comput..

[67]  Jose C. Principe,et al.  A Self-Organizing Temporal Pattern Recognizer with Application to Robot Landmark Recognition , 1996 .

[68]  S. Chiba,et al.  Dynamic programming algorithm optimization for spoken word recognition , 1978 .

[69]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[70]  Erkki Oja,et al.  Neural Networks, Principal Components, and Subspaces , 1989, Int. J. Neural Syst..

[71]  G. Wallis.  Spatio-temporal influences at the neural level of object recognition , 1998 .

[72]  Geoffrey E. Hinton,et al.  The Helmholtz Machine , 1995, Neural Computation.

[73]  Thomas P. Caudell,et al.  A neural architecture for pattern sequence verification through inferencing , 1993, IEEE Trans. Neural Networks.

[74]  Richard M. Golden,et al.  Mathematical Methods for Neural Network Analysis and Design , 1996 .

[75]  Wulfram Gerstner,et al.  Why spikes? Hebbian learning and retrieval of time-resolved excitation patterns , 1993, Biological Cybernetics.

[76]  Claudio M. Privitera,et al.  A neural model for generating and learning a rapid movement sequence , 1996, Biological Cybernetics.

[77]  John G. Taylor,et al.  Self-organization in the time domain , 1998 .

[78]  Jouko Lampinen,et al.  Self-Organizing Maps for Spatial and Temporal AR Models , 1989 .

[79]  Erkki Oja,et al.  Independent component analysis by general nonlinear Hebbian-like learning rules , 1998, Signal Process..

[80]  Zoubin Ghahramani,et al.  A Unifying Review of Linear Gaussian Models , 1999, Neural Computation.

[81]  Aluizio F. R. Araújo,et al.  Nonlinear Modeling of Dynamic Systems with the Self-Organizing Map , 2002, ICANN.

[82]  Alessio Micheli,et al.  Recursive self-organizing network models , 2004, Neural Networks.

[83]  Peter J. Gawthrop,et al.  Neural networks for control systems - A survey , 1992, Autom..

[84]  John G. Taylor,et al.  The temporal Kohönen map , 1993, Neural Networks.

[85]  Lei Xu,et al.  Temporal BYY learning for state space approach, hidden Markov model, and blind source separation , 2000, IEEE Trans. Signal Process..

[86]  Jose C. Principe,et al.  A Spatio-Temporal Memory Based on SOMs with Activity Diffusion , 1999 .

[87]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[88]  Thomas Voegtlin,et al.  Context quantization and contextual self-organizing maps , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[89]  A.F.R. Araujo,et al.  A self-organizing NARX network and its application to prediction of chaotic time series , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[90]  J. L. van Hemmen,et al.  Hebbian learning reconsidered: Representation of static and dynamic objects in associative neural nets , 1989, Biological Cybernetics.

[91]  Jürgen Schmidhuber,et al.  Unsupervised Learning in LSTM Recurrent Neural Networks , 2001, ICANN.

[92]  BART KOSKO,et al.  Bidirectional associative memories , 1988, IEEE Trans. Syst. Man Cybern..

[93]  T J Sejnowski,et al.  Learning viewpoint-invariant face representations from visual experience in an attractor network. , 1998, Network.

[94]  Peter Meinicke.  Unsupervised learning in a generalized regression framework , 2000 .

[95]  Shun-ichi Amari,et al.  Adaptive blind signal processing-neural network approaches , 1998, Proc. IEEE.

[96]  John F. Kolen,et al.  Dynamical Recurrent Networks , 2001 .

[97]  T. Sejnowski,et al.  The predictive brain: temporal coincidence and temporal order in synaptic learning mechanisms. , 1994, Learning & memory.

[98]  Bernhard Schölkopf,et al.  View-Based Cognitive Mapping and Path Planning , 1995, Adapt. Behav..

[99]  Stephen Grossberg,et al.  Working Memory Networks for Learning Temporal Order with Application to Three-Dimensional Visual Object Recognition , 1992, Neural Computation.

[100]  Colin Fyfe,et al.  A temporal model of linear anti-Hebbian learning , 1996, Neural Processing Letters.

[101]  J. Hopfield.  Neurons withgraded response havecollective computational properties likethoseoftwo-state neurons , 1984 .

[102]  Risto Miikkulainen,et al.  SARDNET: A Self-Organizing Feature Map for Sequences , 1994, NIPS.

[103]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[104]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[105]  G. Wallis,et al.  Spatio-temporal influences at the neural level of object recognition. , 1998, Network.

[106]  Peter Tiño,et al.  Learning long-term dependencies in NARX recurrent neural networks , 1996, IEEE Trans. Neural Networks.

[107]  Jürgen Schmidhuber,et al.  Semilinear Predictability Minimization Produces Well-Known Feature Detectors , 1996, Neural Computation.

[108]  Samuel Kaski,et al.  Bibliography of Self-Organizing Map (SOM) Papers: 1981-1997 , 1998 .

[109]  Barbara Hammer,et al.  Unsupervised Recursive Sequence Processing , 2003, ESANN.

[110]  Barak A. Pearlmutter.  Gradient calculations for dynamic recurrent neural networks: a survey , 1995, IEEE Trans. Neural Networks.

[111]  Panos J. Antsaklis,et al.  Neural networks for control systems , 1990, IEEE Trans. Neural Networks.

[112]  James A. Anderson,et al.  Cognitive and psychological computation with neural models , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[113]  Mark D. Plumbley,et al.  Unsupervised neural network learning procedures for feature extraction and classification , 1996, Applied Intelligence.

[114]  J.C. Principe,et al.  Non-linear time series modeling with self-organization feature maps , 1995, Proceedings of 1995 IEEE Workshop on Neural Networks for Signal Processing.

[115]  R. Zemel.  A minimum description length framework for unsupervised learning , 1994 .

[116]  Te-Won Lee,et al.  Blind Separation of Delayed and Convolved Sources , 1996, NIPS.

[117]  Rajesh P. N. Rao,et al.  Predictive Sequence Learning in Recurrent Neocortical Circuits , 1999, NIPS.

[118]  Stephen Grossberg,et al.  The ART of adaptive pattern recognition by a self-organizing neural network , 1988, Computer.

[119]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[120]  John F. Kolen,et al.  Gradient Calculations for Dynamic Recurrent Neural Networks , 2001 .

[121]  Jari Kangas,et al.  Time-delayed self-organizing maps , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[122]  Colin Fyfe,et al.  An extended exploratory projection pursuit network with linear and nonlinear anti-hebbian lateral connections applied to the cocktail party problem , 1997, Neural Networks.

[123]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[124]  Andreas V. M. Herz,et al.  Spatiotemporal association in neural networks , 1998 .

[125]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[126]  Marian Stewart Bartlett,et al.  Face image analysis by unsupervised learning , 2001 .

[127]  Shun-ichi Amari,et al.  Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements , 1972, IEEE Transactions on Computers.

[128]  C. Scheier,et al.  Unsupervised Classiication of Sensory-motor States in a Real World Artifact Using a Temporal Kohonen Map , 1995 .

[129]  Stephen Grossberg,et al.  STORE working memory networks for storage and recall of arbitrary temporal sequences , 1994, Biological Cybernetics.

[130]  Michael A. Arbib,et al.  Timing and chunking in processing temporal order , 1993, IEEE Trans. Syst. Man Cybern..

[131]  Stefan C. Kremer,et al.  Spatiotemporal Connectionist Networks: A Taxonomy and Review , 2001, Neural Computation.

[132]  Samuel Kaski,et al.  Self-Organized Formation of Various Invariant-Feature Filters in the Adaptive-Subspace SOM , 1997, Neural Computation.

[133]  Geoffrey E. Hinton,et al.  GTM through time , 1997 .

[134]  Christopher G. Atkeson,et al.  Implementing projection pursuit learning , 1996, IEEE Trans. Neural Networks.

[135]  Amaury Lendasse,et al.  Time series forecasting with SOM and local non-linear models - Application to the DAX30 index prediction , 2003 .

[136]  J. Príncipe,et al.  Temporal decorrelation using teacher forcing anti-Hebbian learning and its application in adaptive blind source separation , 1996, Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop.

[137]  George Francis Harpur,et al.  Low Entropy Coding with Unsupervised Neural Networks , 1997 .

[138]  DeLiang Wang,et al.  Incremental learning of complex temporal patterns , 1996, IEEE Trans. Neural Networks.

[139]  Erkki Oja,et al.  Independent Component Analysis , 2001 .

[140]  Paul A. Viola,et al.  Empirical Entropy Manipulation for Real-World Problems , 1995, NIPS.

[141]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[142]  Stefan C. Kremer,et al.  On the computational power of Elman-style recurrent networks , 1995, IEEE Trans. Neural Networks.

[143]  B Brückner,et al.  A MODIFIED HYPERMAP ARCHITECTURE FOR CLASSIFICATION OF BIOLOGICAL SIGNALS , 1992 .

[144]  Sepp Hochreiter,et al.  The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions , 1998, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[145]  Jari Kangas.  Phoneme recognition using time-dependent versions of self-organizing maps , 1991, [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing.

[146]  James A. Reggia,et al.  Temporally Asymmetric Learning Supports Sequence Processing in Multi-Winner Self-Organizing Maps , 2004, Neural Computation.

[147]  G. Tesauro,et al.  Simple neural models of classical conditioning , 1986, Biological Cybernetics.

[148]  Wolfgang Maass,et al.  Fast Sigmoidal Networks via Spiking Neurons , 1997, Neural Computation.

[149]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[150]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.

[151]  Thomas Martinetz,et al.  'Neural-gas' network for vector quantization and its application to time-series prediction , 1993, IEEE Trans. Neural Networks.

[152]  Stephen Grossberg,et al.  Fuzzy ART: Fast stable learning and categorization of analog patterns by an adaptive resonance system , 1991, Neural Networks.

[153]  F. H. Lopes da Silva,et al.  Spatio-temporal models in biological and artificial systems , 1997 .

[154]  José Carlos Príncipe,et al.  Generalized anti-Hebbian learning for source separation , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[155]  Bernd Brückner,et al.  Improvements of the modified hypermap architecture for speech recognition , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[156]  Masafumi Hagiwara.  Time-Delay ART for spatio-temporal patterns , 1994, Neurocomputing.

[157]  Michael A. Arbib,et al.  Complex temporal sequence learning based on short-term memory , 1990 .

[158]  Michel Verleysen,et al.  Forecasting electricity consumption using nonlinear projection and self-organizing maps , 2002, Neurocomputing.

[159]  Young-Seuk Park,et al.  Determining temporal pattern of community dynamics by using unsupervised learning algorithms , 2000 .

[160]  Mathias Bode,et al.  Wave Propagation in Self-Organizing Feature Maps as a Means for the Representation of Temporal Sequences , 1997, ICANN.

[161]  Gillian M. Hayes,et al.  Object localisation in 2D images using a temporal Kohonen network , 1997 .

[162]  Kanter,et al.  Temporal association in asymmetric neural networks. , 1986, Physical review letters.

[163]  A G Barto,et al.  Toward a modern theory of adaptive networks: expectation and prediction. , 1981, Psychological review.

[164]  George H. Mealy,et al.  A method for synthesizing sequential circuits , 1955 .

[165]  A. E. Maxwell,et al.  Factor Analysis as a Statistical Method. , 1964 .

[166]  Aapo Hyvärinen,et al.  Survey on Independent Component Analysis , 1999 .

[167]  Sen Song,et al.  Temporally asymmetric Hebbian learning and neuronal response variability , 2000, Neurocomputing.

[168]  Panu Somervuo,et al.  Self-Organizing Maps and Learning Vector Quantization for Feature Sequences , 1999, Neural Processing Letters.

[169]  Edward F. Moore,et al.  Gedanken-Experiments on Sequential Machines , 1956 .

[170]  A. PearlmutterB..  Gradient calculations for dynamic recurrent neural networks , 1995 .

[171]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[172]  James V. Stone.  Learning Perceptually Salient Visual Parameters Using Spatiotemporal Smoothness Constraints , 1996, Neural Computation.

[173]  S Grossberg,et al.  Masking fields: a massively parallel neural architecture for learning, recognizing, and predicting multiple groupings of patterned data. , 1987, Applied optics.

[174]  S. Grossberg,et al.  Pattern Recognition by Self-Organizing Neural Networks , 1991 .

[175]  Wolfgang Maass,et al.  On the relevance of time in neural computation and learning , 2001, Theor. Comput. Sci..

[176]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[177]  Christopher M. Bishop,et al.  GTM: The Generative Topographic Mapping , 1998, Neural Computation.