Memory capacity of networks with stochastic binary synapses

In standard attractor neural network models, specific patterns of activity are stored in the synaptic matrix, so that they become fixed point attractors of the network dynamics. The storage capacity of such networks has been quantified in two ways: the maximal number of patterns that can be stored, and the stored information measured in bits per synapse. In this paper, we compute both quantities in fully connected networks of N binary neurons with binary synapses, storing patterns with coding level f , in the large N and sparse coding limits (N??, f?0). We also derive finite-size corrections that accurately reproduce the results of simulations in networks of tens of thousands of neurons. These methods are applied to three different scenarios: (1) the classic Willshaw model, (2) networks with stochastic learning in which patterns are shown only once (one shot learning), (3) networks with stochastic learning in which patterns are shown multiple times. The storage capacities are optimized over network parameters, which allows us to compare the performance of the different models. We show that finite-size effects strongly reduce the capacity, even for networks of realistic sizes. We discuss the implications of these results for memory storage in the hippocampus and cerebral cortex. Citation: Dubreuil AM, Amit Y, Brunel N (2014) Memory Capacity of Networks with Stochastic Binary Synapses. PLoS Comput Biol 10(8): e1003727. doi:10.1371/ journal.pcbi.1003727 Editor: Claus C. Hilgetag, Hamburg University, Germany Received February 12, 2014; Accepted May 30, 2014; Published August 7, 2014 Copyright: 2014 Dubreuil et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: AMD is supported by a grant from the French Ministry of Higher Education and Research. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing Interests: The authors have declared that no competing interests exist. * Email: nbrunel@galton.uchicago.edu

[1]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[2]  Nicolas Brunel,et al.  Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons , 2000, Journal of Computational Neuroscience.

[3]  T. Sejnowski,et al.  Storing covariance with nonlinearly interacting neurons , 1977, Journal of mathematical biology.

[4]  J. Montgomery,et al.  Discrete synaptic states define a major mechanism of synapse plasticity , 2004, Trends in Neurosciences.

[5]  Nicolas Brunel,et al.  NETWORK MODELS OF MEMORY , 2004 .

[6]  S. Wang,et al.  Graded bidirectional synaptic plasticity is composed of switch-like unitary events. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[7]  Mark C. W. van Rossum,et al.  Optimal Learning Rules for Discrete Synapses , 2008, PLoS Comput. Biol..

[8]  Alan Fine,et al.  Expression of Long-Term Plasticity at Individual Synapses in Hippocampus Is Graded, Bidirectional, and Mainly Presynaptic: Optical Quantal Analysis , 2009, Neuron.

[9]  J. Fuster.  Memory in the cerebral cortex , 1994 .

[10]  Yali Amit,et al.  Capacity analysis in multi-state synaptic models: a retrieval probability perspective , 2011, Journal of Computational Neuroscience.

[11]  Yali Amit,et al.  Memory Capacity of Networks with Stochastic Binary Synapses , 2013, PLoS Comput. Biol..

[12]  Günther Palm,et al.  Memory Capacities for Synaptic and Structural Plasticity G ¨ Unther Palm , 2022 .

[13]  A I I,et al.  Associative memory : on the ( puzzling ) sparse coding limit , 1990 .

[14]  Sompolinsky,et al.  Neural networks with nonlinear synapses and a static noise. , 1986, Physical review. A, General physics.

[15]  Y. Miyashita,et al.  Neuronal correlate of pictorial short-term memory in the primate temporal cortexYasushi Miyashita , 1988, Nature.

[16]  Y. Miyashita.  Neuronal correlate of visual associative long-term memory in the primate temporal cortex , 1988, Nature.

[17]  Haim Sompolinsky,et al.  Course 9 - Irregular Activity in Large Networks of Neurons , 2005 .

[18]  D. Amit,et al.  Quantitative study of attractor neural networks retrieving at low spike rates: II. Low-rate retrieval in symmetric networks , 1991 .

[19]  Y. Loewenstein,et al.  Multiplicative Dynamics Underlie the Emergence of the Log-Normal Distribution of Spine Sizes in the Neocortex In Vivo , 2011, The Journal of Neuroscience.

[20]  R. Kempter,et al.  Sparseness constrains the prolongation of memory lifetime via synaptic metaplasticity. , 2008, Cerebral cortex.

[21]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[22]  G. E. Alexander,et al.  Neuron Activity Related to Short-Term Memory , 1971, Science.

[23]  Jacek Iwanski.  Replica symmetry breaking calculation of the storage capacity of neural networks with discrete couplings , 1994 .

[24]  Daniel J. Amit,et al.  Learning in Neural Networks with Material Synapses , 1994, Neural Computation.

[25]  G. Parisi.  A memory which forgets , 1986 .

[26]  Stanislas Dehaene,et al.  Networks of Formal Neurons and Memory Palimpsests , 1986 .

[27]  Haim Sompolinsky,et al.  Chaotic Balanced State in a Model of Cortical Circuits , 1998, Neural Computation.

[28]  N Brunel,et al.  Slow stochastic Hebbian learning of classes of stimuli in a recurrent neural network. , 1998, Network.

[29]  Daniel J. Amit,et al.  Modeling brain function: the world of attractor neural networks, 1st Edition , 1989 .

[30]  P. Dayan,et al.  Off-line replay maintains declarative memories in a model of hippocampal-neocortical interactions , 2004, Nature Neuroscience.

[31]  H. Gutfreund,et al.  Capacity of neural networks with discrete synaptic couplings , 1990 .

[32]  P Alvarez,et al.  Memory consolidation and the medial temporal lobe: a simple network model. , 1994, Proceedings of the National Academy of Sciences of the United States of America.

[33]  H. Sompolinsky,et al.  Chaos in Neuronal Networks with Balanced Excitatory and Inhibitory Activity , 1996, Science.

[34]  K. Nakamura,et al.  Mnemonic firing of neurons in the monkey temporal pole during a visual recognition memory task. , 1995, Journal of neurophysiology.

[35]  N. Brunel.  Storage capacity of neural networks: effect of the fluctuations of the number of active neurons per memory , 1994 .

[36]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[37]  P. Goldman-Rakic.  Cellular basis of working memory , 1995, Neuron.

[38]  Y. Miyashita.  Inferior temporal cortex: where visual perception meets memory. , 1993, Annual review of neuroscience.

[39]  Peter E. Latham,et al.  A Balanced Memory Network , 2007, PLoS Comput. Biol..

[40]  D. Amit,et al.  Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. , 1997, Cerebral cortex.

[41]  Yali Amit,et al.  Precise Capacity Analysis in Binary Networks with Multiple Coding Level Inputs , 2010, Neural Computation.

[42]  J. Hopfield,et al.  All-or-none potentiation at CA3-CA1 synapses. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[43]  Stefano Fusi,et al.  Efficient Partitioning of Memory Systems and Its Importance for Memory Consolidation , 2013, PLoS Comput. Biol..

[44]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[45]  E. Gardner.  The space of interactions in neural network models , 1988 .

[46]  M. Tsodyks.  ASSOCIATIVE MEMORY IN NEURAL NETWORKS WITH BINARY SYNAPSES , 1990 .