Memory Capacities for Synaptic and Structural Plasticity G ¨ Unther Palm

Neural associative networks with plastic synapses have been proposed as computational models of brain functions and also for applications such as pattern recognition and information retrieval. To guide biological models and optimize technical applications, several definitions of memory capacity have been used to measure the efficiency of associative memory. Here we explain why the currently used performance measures bias the comparison between models and cannot serve as a theoretical benchmark. We introduce fair measures for information-theoretic capacity in associative memory that also provide a theoretical benchmark. In neural networks, two types of manipulating synapses can be discerned: synaptic plasticity, the change in strength of existing synapses, and structural plasticity, the creation and pruning of synapses. One of the new types of memory capacity we introduce permits quantifying how structural plasticity can increase the network efficiency by compressing the network structure, for example, by pruning unused synapses. Specifically, we analyze operating regimes in the Willshaw model in which structural plasticity can compress the network structure and push performance to the theoretical benchmark. The amount C of information stored in each synapse can scale with the logarithm of the network size rather than being constant, as in classical Willshaw and Hopfield nets ( ln 2 0.7). Further, the review contains novel technical material: a capacity analysis of the Willshaw model that rigorously controls for the level of retrieval quality, an analysis for memories with a nonconstant number of active units (where C 1eln 2 0.53), and the analysis of the computational complexity of associative memories with and without network compression.

[1]  Günther Palm,et al.  Improved bidirectional retrieval of sparse patterns stored by Hebbian learning , 1999, Neural Networks.

[2]  Peter E. Latham,et al.  Computing and Stability in Cortical Networks , 2004, Neural Computation.

[3]  Peter Redgrave,et al.  Basal Ganglia , 2020, Encyclopedia of Autism Spectrum Disorders.

[4]  S. Golomb.  Run-length encodings. , 1966 .

[5]  Günther Palm,et al.  Controlling the Speed of Synfire Chains , 1996, ICANN.

[6]  M. Opper,et al.  Distribution of Internal Fields and Dynamics of Neural Networks , 1990 .

[7]  Sompolinsky,et al.  Information storage in neural networks with low levels of activity. , 1987, Physical review. A, General physics.

[8]  Andreas Knoblauch,et al.  Neural Associative Memory and the Willshaw--Palm Probability Distribution , 2008, SIAM J. Appl. Math..

[9]  G. Shepherd.  The Synaptic Organization of the Brain , 1979 .

[10]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[11]  Herman H. Goldstine,et al.  Preliminary discussion of the logical design of an electronic computing instrument (1946) , 1989 .

[12]  H. Barlow,et al.  Single Units and Sensation: A Neuron Doctrine for Perceptual Psychology? , 1972, Perception.

[13]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[14]  Solomon W. Golomb,et al.  Run-length encodings (Corresp.) , 1966, IEEE Trans. Inf. Theory.

[15]  Diego Guidolin,et al.  Central Nervous System and Computation , 2011, The Quarterly Review of Biology.

[16]  Edmund T. Rolls,et al.  What determines the capacity of autoassociative memories in the brain? Network , 1991 .

[17]  F. Frances Yao,et al.  Multi-index hashing for information retrieval , 1994, Proceedings 35th Annual Symposium on Foundations of Computer Science.

[18]  Friedrich T. Sommer,et al.  Associative Data Storage and Retrieval in Neural Networks , 1996 .

[19]  David Willshaw,et al.  Capacity and information efficiency of the associative net , 1997 .

[20]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[21]  Karl Steinbuch,et al.  Die Lernmatrix , 2004, Kybernetik.

[22]  A A Frolov,et al.  Informational characteristics of neural networks capable of associative learning based on Hebbian plasticity , 1993 .

[23]  H. Fitz,et al.  Connectionist Models of Behavior and Cognition II. Proceedings of the 11th Neural Computation and Psychology Workshop. , 2009 .

[24]  W. Precht.  The synaptic organization of the brain G.M. Shepherd, Oxford University Press (1975). 364 pp., £3.80 (paperback) , 1976, Neuroscience.

[25]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[26]  A. Coolen.  Statistical Mechanics of Recurrent Neural Networks I. Statics , 2000, cond-mat/0006010.

[27]  Mohamad H. Hassoun,et al.  An RCE-based Associative Memory with Application to Human Face Recognition , 2006, Neural Processing Letters.

[28]  Professor Moshe Abeles,et al.  Local Cortical Circuits , 1982, Studies of Brain Function.

[29]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[30]  Richard W. Prager,et al.  The modified Kanerva model for automatic speech recognition , 1989 .

[31]  C. Koch,et al.  Invariant visual representation by single neurons in the human brain , 2005, Nature.

[32]  Andreas Wichert,et al.  Tree-like hierarchical associative memory structures , 2011, Neural Networks.

[33]  W. Little.  The existence of persistent states in the brain , 1974 .

[34]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[35]  F. Engert,et al.  Dendritic spine changes associated with hippocampal long-term synaptic plasticity , 1999, Nature.

[36]  D. Hammerstrom,et al.  A VLSI architecture for high-performance, low-cost, on-chip learning , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[37]  Glenn C. Turner,et al.  Oscillations and Sparsening of Odor Representations in the Mushroom Body , 2002, Science.

[38]  Ángel Sánchez,et al.  Learning sequences of sparse correlated patterns using small-world attractor neural networks: An application to traffic videos , 2011, Neurocomputing.

[39]  Günther Palm,et al.  On Associative Memories , 1987 .

[40]  S. Laughlin,et al.  An Energy Budget for Signaling in the Grey Matter of the Brain , 2001, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[41]  E. Gardner,et al.  Optimal storage properties of neural network models , 1988 .

[42]  David A. Huffman,et al.  A method for the construction of minimum-redundancy codes , 1952, Proceedings of the IRE.

[43]  William R. Softky,et al.  The highly irregular firing of cortical cells is inconsistent with temporal integration of random EPSPs , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[44]  Mike Butts,et al.  FPGA Implementation of Very Large Associative Memories , 2006 .

[45]  Rafal Bogacz,et al.  Model of Familiarity Discrimination in the Perirhinal Cortex , 2004, Journal of Computational Neuroscience.

[46]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[47]  Andreas Knoblauch,et al.  Optimal Matrix Compression Yields Storage Capacity 1 for Binary Willshaw Associative Memory , 2003, ICANN.

[48]  G. Palm,et al.  On associative memory , 2004, Biological Cybernetics.

[49]  P. Dayan,et al.  Optimising synaptic learning rules in linear associative memories , 1991, Biological Cybernetics.

[50]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[51]  A I I,et al.  Associative memory : on the ( puzzling ) sparse coding limit , 1990 .

[52]  Günther Palm,et al.  Information storage and effective data retrieval in sparse matrices , 1989, Neural Networks.

[53]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[54]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[55]  David J. Willshaw,et al.  Improving recall from an associative memory , 1995, Biological Cybernetics.

[56]  V. Braitenberg.  Cell Assemblies in the Cerebral Cortex , 1978 .

[57]  Andreas Knoblauch,et al.  Pattern separation and synchronization in spiking associative memories and visual areas , 2001, Neural Networks.

[58]  J. A. Anderson,et al.  A memory storage model utilizing spatial correlation functions , 1968, Kybernetik.

[59]  D Marr,et al.  Simple memory: a theory for archicortex. , 1971, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[60]  Teuvo Kohonen,et al.  Associative memory. A system-theoretical approach , 1977 .

[61]  KnoblauchAndreas,et al.  Memory capacities for synaptic and structural plasticity , 2010 .

[62]  Vittorio Dante,et al.  A VLSI recurrent network of integrate-and-fire neurons connected by plastic synapses with long-term memory , 2003, IEEE Trans. Neural Networks.

[63]  Dmitri A. Rachkovskij,et al.  Binding and Normalization of Binary Sparse Distributed Representations by Context-Dependent Thinning , 2001, Neural Computation.

[64]  M Abeles,et al.  Spatio-temporal firing patterns in the frontal cortex of behaving monkeys , 1996, Journal of Physiology-Paris.

[65]  K. D. Punta,et al.  An ultra-sparse code underlies the generation of neural sequences in a songbird , 2002 .

[66]  J. Knott.  The organization of behavior: A neuropsychological theory , 1951 .

[67]  Sompolinsky,et al.  Willshaw model: Associative memory with sparse coding and low firing rates. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[68]  J. Albus.  A Theory of Cerebellar Function , 1971 .

[69]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[70]  T. M. Mayhew,et al.  Anatomy of the Cortex: Statistics and Geometry. , 1991 .

[71]  C. Koch,et al.  Sparse Representation in the Human Medial Temporal Lobe , 2006, The Journal of Neuroscience.

[72]  Gilles Laurent,et al.  Olfactory network dynamics and the coding of multidimensional signals , 2002, Nature Reviews Neuroscience.

[73]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[74]  H. Cline,et al.  In vivo observations of timecourse and distribution of morphological dynamics in Xenopus retinotectal axon arbors. , 1996, Journal of neurobiology.

[75]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[76]  G Palm,et al.  Computing with neural networks. , 1987, Science.

[77]  Jean-Pierre Nadal,et al.  Information storage in sparsely coded memory nets , 1990 .

[78]  C. Woolley,et al.  Structural plasticity of dendrites , 2012 .

[79]  Bernhard Hellwig,et al.  A quantitative analysis of the local connectivity between pyramidal neurons in layers 2/3 of the rat visual cortex , 2000, Biological Cybernetics.

[80]  Andreas Knoblauch,et al.  Synchronization and pattern separation in spiking associative memories and visual cortical areas , 2004 .

[81]  D. Chklovskii,et al.  Geometry and Structural Plasticity of Synaptic Connectivity , 2002, Neuron.

[82]  A.C.C. Coolen,et al.  Chapter 15 Statistical mechanics of recurrent neural networks II — Dynamics , 2000, cond-mat/0006011.

[83]  P. Lennie.  The Cost of Cortical Computation , 2003, Current Biology.

[84]  E. Rolls.  A theory of hippocampal function in memory , 1996, Hippocampus.

[85]  Ad Aertsen,et al.  Stable propagation of synchronous spiking in cortical neural networks , 1999, Nature.

[86]  E Fransén,et al.  A model of cortical associative memory based on a horizontal network of connected columns. , 1998, Network.

[87]  L. Abbott,et al.  Cascade Models of Synaptically Stored Memories , 2005, Neuron.

[88]  Stephen A. Ritz,et al.  Distinctive features, categorical perception, and probability learning: some applications of a neural model , 1977 .

[89]  Andreas Knoblauch,et al.  Neural Associative Memory with Optimal Bayesian Learning , 2011, Neural Computation.

[90]  Joseph E LeDoux,et al.  Structural plasticity and memory , 2004, Nature Reviews Neuroscience.

[91]  E. Vaadia,et al.  Spatiotemporal firing patterns in the frontal cortex of behaving monkeys. , 1993, Journal of neurophysiology.

[92]  Günther Palm,et al.  Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states , 1992 .

[93]  Andreas Knoblauch,et al.  Neural associative memory for brain modeling and information retrieval , 2005, Inf. Process. Lett..

[94]  Prof. Dr. Valentino Braitenberg,et al.  Anatomy of the Cortex , 1991, Studies of Brain Function.

[95]  A. R. Gardner-Medwin.  The recall of events through the learning of associations between their parts , 1976, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[96]  Franz J. Kurfess,et al.  Information storage capacity of incompletely connected associative memories , 1998, Neural Networks.

[97]  Rafal Bogacz,et al.  Comparison of computational models of familiarity discrimination in the perirhinal cortex , 2003, Hippocampus.

[98]  D. Marr.  A theory of cerebellar cortex , 1969, The Journal of physiology.

[99]  James A. Anderson,et al.  The BSB model: a simple nonlinear autoassociative neural network , 1993 .

[100]  A. Knoblauch,et al.  THE ROLE OF STRUCTURAL PLASTICITY AND SYNAPTIC CONSOLIDATION FOR MEMORY AND AMNESIA IN A MODEL OF CORTICO-HIPPOCAMPAL INTERPLAY , 2009 .

[101]  Shun-ichi Amari,et al.  Characteristics of sparsely encoded associative memory , 1989, Neural Networks.

[102]  Bartlett W. Mel,et al.  Impact of Active Dendrites and Structural Plasticity on the Memory Capacity of Neural Tissue , 2001, Neuron.

[103]  Peter Dayan,et al.  Bayesian retrieval in associative memories with storage errors , 1998, IEEE Trans. Neural Networks.

[104]  Gèunther Palm,et al.  Neural Assemblies: An Alternative Approach to Artificial Intelligence , 1982 .

[105]  Jay Buckinghamts.  On setting unit thresholds in an incompletely connected associative net , 1993 .

[106]  A. Grinvald,et al.  Dynamics of Ongoing Activity: Explanation of the Large Variability in Evoked Cortical Responses , 1996, Science.

[107]  Friedemann Pulvermüller,et al.  The Neuroscience of Language: On Brain Circuits of Words and Serial Order , 2003 .

[108]  Moshe Abeles,et al.  Memory Capacity of Balanced Networks , 2005, Neural Computation.

[109]  Terrence J Sejnowski,et al.  Communication in Neuronal Networks , 2003, Science.

[110]  David Willshaw,et al.  Performance characteristics of the associative net , 1992 .

[111]  Ulrich Rückert,et al.  Mixed Mode VLSI Implementation of a Neural Associative Memory , 1999 .

[112]  Günther Palm,et al.  Iterative retrieval of sparsely coded associative memory patterns , 1996, Neural Networks.

[113]  Garrett E. Alexander.  Basal ganglia , 1998 .

[114]  Martin Rehn,et al.  Storing and restoring visual input with collaborative rank coding and associative memory , 2006, Neurocomputing.

[115]  J. Buckingham,et al.  Delicate nets, faint recollections : a study of partially connected associative network memories , 1992 .

[116]  Roland Heim,et al.  Theoretical Approaches to Complex Systems , 1978 .