Spatio–Temporal Memories for Machine Learning: A Long-Term Memory Organization

Design of artificial neural structures capable of reliable and flexible long-term spatio-temporal memory is of paramount importance in machine intelligence. To this end, we propose a novel, biologically inspired, long-term memory (LTM) architecture. We intend to use it as a building block of a neuron-level architecture that is able to mimic natural intelligence through learning, anticipation, and goal-driven behavior. A mutual input enhancement and blocking structure is proposed, and its operation is discussed in detail. The paper focuses on a hierarchical memory organization, storage, recognition, and recall mechanisms. Simulation results of the proposed memory show its effectiveness, adaptability, and robustness. Accuracy of the proposed method is compared to other methods including Levenshtein distance method and a Markov chain.

[1]  E R Kandel,et al.  Consolidation of learning strategies during spatial working memory task requires protein synthesis in the prefrontal cortex , 2007, Proceedings of the National Academy of Sciences.

[2]  C. Lee Giles,et al.  Sequence learning: from recognition and prediction to sequential decision making , 2001, IEEE Intelligent Systems.

[3]  Graham Rawlinson,et al.  The Significance of Letter Position in Word Recognition , 2007, IEEE Aerospace and Electronic Systems Magazine.

[4]  Stephen Grossberg,et al.  A laminar cortical model of stereopsis and 3D surface perception: closure and da Vinci stereopsis. , 2004, Spatial vision.

[5]  S. Grossberg.  How does the cerebral cortex work? Learning, attention, and grouping by the laminar circuits of visual cortex. , 1999, Spatial vision.

[6]  G. Müller,et al.  Experimentelle Beiträge zur Lehre vom Gedächtniss , 1900 .

[7]  D. George,et al.  Hierarchical Temporal Memory Concepts , Theory , and Terminology , 2006 .

[8]  Michael A. Arbib,et al.  Complex temporal sequence learning based on short-term memory , 1990 .

[9]  J. Starzyk.  Motivation in Embodied Intelligence , 2008 .

[10]  S. Grossberg,et al.  A laminar cortical model of stereopsis and three-dimensional surface perception , 2003, Vision Research.

[11]  Denis Paré,et al.  Lasting increases in basolateral amygdala activity after emotional arousal: implications for facilitated consolidation of emotional memories. , 2005, Learning & memory.

[12]  L. Squire,et al.  100 years of consolidation--remembering Müller and Pilzecker. , 1999, Learning & memory.

[13]  Salah Bourennane,et al.  Unsupervised image segmentation based on high-order hidden Markov chains [radar imaging examples] , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[14]  D. Stegeman,et al.  Solution Methods of Electrical Field Problems in Physiology , 1982, IEEE Transactions on Biomedical Engineering.

[15]  Otávio Augusto S. Carpinteiro.  A Hierarchical Self-Organizing Map Model for Sequence Recognition , 2004, Neural Processing Letters.

[16]  James L McGaugh,et al.  Emotional arousal and enhanced amygdala activity: new evidence for the old perseveration-consolidation hypothesis. , 2005, Learning & memory.

[17]  Michael I. Jordan.  Attractor dynamics and parallelism in a connectionist sequential machine , 1990 .

[18]  Stefano Nolfi,et al.  Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems , 1998, Neural Networks.

[19]  Marek R. Ogiela,et al.  Why Automatic Understanding? , 2007, ICANNGA.

[20]  Michael K. Ng,et al.  Higher‐order Markov chain models for categorical data sequences * , 2004 .

[21]  J. Hawkins,et al.  Why Can't a Computer be more Like a Brain? , 2007, IEEE Spectrum.

[22]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[23]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.

[24]  Salah Bourennane,et al.  Unsupervised image segmentation based on high-order hidden Markov chain , 2004 .

[25]  Jörg Lücke,et al.  Rapid Processing and Unsupervised Learning in a Model of the Cortical Macrocolumn , 2004, Neural Computation.

[26]  Jun Tani,et al.  Learning to generate articulated behavior through the bottom-up and the top-down interaction processes , 2003, Neural Networks.

[27]  Joy Bose,et al.  An associative memory for the on-line recognition and prediction of temporal sequences , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[28]  Michael A. Arbib,et al.  Timing and chunking in processing temporal order , 1993, IEEE Trans. Syst. Man Cybern..

[29]  Stefan C. Kremer,et al.  Spatiotemporal Connectionist Networks: A Taxonomy and Review , 2001, Neural Computation.

[30]  J. Desmond,et al.  Load-Dependent Roles of Frontal Brain Regions in the Maintenance of Working Memory , 1999, NeuroImage.

[31]  Henrik Jacobsson,et al.  Rule Extraction from Recurrent Neural Networks: ATaxonomy and Review , 2005, Neural Computation.

[32]  V. Mountcastle.  Perceptual Neuroscience: The Cerebral Cortex , 1998 .

[33]  R. Henson.  Short-Term Memory for Serial Order: The Start-End Model , 1998, Cognitive Psychology.

[34]  R. Gerard.  Physiology and psychiatry. , 1949, The American journal of psychiatry.

[35]  Mounir Boukadoum,et al.  A sequential dynamic heteroassociative memory for multistep pattern recognition and one-to-many association , 2006, IEEE Transactions on Neural Networks.

[36]  J. D. McGaugh.  Memory--a century of consolidation. , 2000, Science.

[37]  DeLiang Wang,et al.  Incremental learning of complex temporal patterns , 1996, IEEE Trans. Neural Networks.

[38]  Haibo He,et al.  Anticipation-Based Temporal Sequences Learning in Hierarchical Structure , 2007, IEEE Transactions on Neural Networks.

[39]  Lipo Wang,et al.  Multi-associative neural networks and their applications to learning and retrieving complex spatio-temporal sequences , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[40]  Lipo Wang,et al.  Learning and retrieving spatio-temporal sequences with any static associative neural network , 1998 .

[41]  DeLiang Wang,et al.  Anticipation-based temporal pattern generation , 1995, IEEE Trans. Syst. Man Cybern..

[42]  J. Hawkins,et al.  On Intelligence , 2004 .

[43]  J. Haldane.  The interaction of nature and nurture. , 1946, Annals of eugenics.

[44]  R. O’Reilly,et al.  Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain , 2000 .

[45]  J. Pollack.  The Induction of Dynamical Recognizers , 1996, Machine Learning.

[46]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[47]  J. Jay Todd,et al.  Capacity limit of visual short-term memory in human posterior parietal cortex , 2004, Nature.

[48]  N. Burgess,et al.  Computational models of working memory: putting long-term memory into context , 2005, Trends in Cognitive Sciences.

[49]  Morris Moscovitch,et al.  Neuroimaging the Serial Position Curve , 2005, Psychological science.

[50]  David D. Vogel,et al.  A neural network model of memory and higher cognitive functions. , 2005, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[51]  Leslie G. Ungerleider,et al.  A neural system for human visual working memory. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[52]  Aluizio F. R. Araújo,et al.  Context in temporal sequence processing: a self-organizing approach and its application to robotics , 2002, IEEE Trans. Neural Networks.