Memory Capacity of Networks with Stochastic Binary Synapses

In standard attractor neural network models, specific patterns of activity are stored in the synaptic matrix, so that they become fixed point attractors of the network dynamics. The storage capacity of such networks has been quantified in two ways: the maximal number of patterns that can be stored, and the stored information measured in bits per synapse. In this paper, we compute both quantities in fully connected networks of N binary neurons with binary synapses, storing patterns with coding level , in the large and sparse coding limits (). We also derive finite-size corrections that accurately reproduce the results of simulations in networks of tens of thousands of neurons. These methods are applied to three different scenarios: (1) the classic Willshaw model, (2) networks with stochastic learning in which patterns are shown only once (one shot learning), (3) networks with stochastic learning in which patterns are shown multiple times. The storage capacities are optimized over network parameters, which allows us to compare the performance of the different models. We show that finite-size effects strongly reduce the capacity, even for networks of realistic sizes. We discuss the implications of these results for memory storage in the hippocampus and cerebral cortex.

[1]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[2]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[3]  G. E. Alexander,et al.  Neuron Activity Related to Short-Term Memory , 1971, Science.

[4]  T. Sejnowski,et al.  Storing covariance with nonlinearly interacting neurons , 1977, Journal of mathematical biology.

[5]  H. Sompolinsky,et al.  Chaos in Neuronal Networks with Balanced Excitatory and Inhibitory Activity , 1996, Science.

[6]  K. Nakamura,et al.  Mnemonic firing of neurons in the monkey temporal pole during a visual recognition memory task. , 1995, Journal of neurophysiology.

[7]  Nicolas Brunel,et al.  Phase diagrams of sparsely connected networks of excitatory and inhibitory spiking neurons , 2000, Neurocomputing.

[8]  Sompolinsky,et al.  Neural networks with nonlinear synapses and a static noise. , 1986, Physical review. A, General physics.

[9]  Y. Miyashita,et al.  Neuronal correlate of pictorial short-term memory in the primate temporal cortexYasushi Miyashita , 1988, Nature.

[10]  N. Brunel.  Storage capacity of neural networks: effect of the fluctuations of the number of active neurons per memory , 1994 .

[11]  Daniel J. Amit,et al.  Learning in Neural Networks with Material Synapses , 1994, Neural Computation.

[12]  E. Gardner.  The space of interactions in neural network models , 1988 .

[13]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Haim Sompolinsky,et al.  Chaotic Balanced State in a Model of Cortical Circuits , 1998, Neural Computation.

[15]  W. Krauth,et al.  Storage capacity of memory networks with binary couplings , 1989 .

[16]  Werner Krauth,et al.  Critical storage capacity of the J = ± 1 neural network , 1989 .

[17]  Günther Palm,et al.  Memory Capacities for Synaptic and Structural Plasticity G ¨ Unther Palm , 2022 .

[18]  D. Amit,et al.  Quantitative study of attractor neural networks retrieving at low spike rates: II. Low-rate retrieval in symmetric networks , 1991 .

[19]  Stanislas Dehaene,et al.  Networks of Formal Neurons and Memory Palimpsests , 1986 .

[20]  Nicolas Brunel,et al.  Efficient supervised learning in networks with binary synapses , 2007, BMC Neuroscience.

[21]  Nicolas Brunel,et al.  Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons , 2000, Journal of Computational Neuroscience.

[22]  J. Montgomery,et al.  Discrete synaptic states define a major mechanism of synapse plasticity , 2004, Trends in Neurosciences.

[23]  M. Tsodyks.  ASSOCIATIVE MEMORY IN NEURAL NETWORKS WITH BINARY SYNAPSES , 1990 .

[24]  R. Kempter,et al.  Sparseness constrains the prolongation of memory lifetime via synaptic metaplasticity. , 2008, Cerebral cortex.

[25]  S. Wang,et al.  Graded bidirectional synaptic plasticity is composed of switch-like unitary events. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[26]  Alan Fine,et al.  Expression of Long-Term Plasticity at Individual Synapses in Hippocampus Is Graded, Bidirectional, and Mainly Presynaptic: Optical Quantal Analysis , 2009, Neuron.

[27]  Haim Sompolinsky,et al.  Course 9 - Irregular Activity in Large Networks of Neurons , 2005 .

[28]  Yali Amit,et al.  Precise Capacity Analysis in Binary Networks with Multiple Coding Level Inputs , 2010, Neural Computation.

[29]  J. Hopfield,et al.  All-or-none potentiation at CA3-CA1 synapses. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[30]  Stefano Fusi,et al.  Efficient Partitioning of Memory Systems and Its Importance for Memory Consolidation , 2013, PLoS Comput. Biol..

[31]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[32]  P Alvarez,et al.  Memory consolidation and the medial temporal lobe: a simple network model. , 1994, Proceedings of the National Academy of Sciences of the United States of America.

[33]  J. Nadal,et al.  Associative memory: on the (puzzling) sparse coding limit , 1991 .

[34]  Y. Miyashita.  Neuronal correlate of visual associative long-term memory in the primate temporal cortex , 1988, Nature.

[35]  J. Fuster.  Memory in the cerebral cortex , 1994 .

[36]  Mark C. W. van Rossum,et al.  Optimal Learning Rules for Discrete Synapses , 2008, PLoS Comput. Biol..

[37]  Yali Amit,et al.  Capacity analysis in multi-state synaptic models: a retrieval probability perspective , 2011, Journal of Computational Neuroscience.

[38]  P. Goldman-Rakic.  Cellular basis of working memory , 1995, Neuron.

[39]  Y. Miyashita.  Inferior temporal cortex: where visual perception meets memory. , 1993, Annual review of neuroscience.

[40]  Peter E. Latham,et al.  A Balanced Memory Network , 2007, PLoS Comput. Biol..

[41]  D. Amit,et al.  Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. , 1997, Cerebral cortex.

[42]  Y. Loewenstein,et al.  Multiplicative Dynamics Underlie the Emergence of the Log-Normal Distribution of Spine Sizes in the Neocortex In Vivo , 2011, The Journal of Neuroscience.

[43]  N Brunel,et al.  Slow stochastic Hebbian learning of classes of stimuli in a recurrent neural network. , 1998, Network.

[44]  Daniel J. Amit,et al.  Modeling brain function: the world of attractor neural networks, 1st Edition , 1989 .

[45]  P. Dayan,et al.  Off-line replay maintains declarative memories in a model of hippocampal-neocortical interactions , 2004, Nature Neuroscience.

[46]  H. Gutfreund,et al.  Capacity of neural networks with discrete synaptic couplings , 1990 .

[47]  G. Parisi.  A memory which forgets , 1986 .