Mean Field and Capacity in Realistic Networks of Spiking Neurons Storing Sparsely Coded Random Memories

Mean-field (MF) theory is extended to realistic networks of spiking neurons storing in synaptic couplings of randomly chosen stimuli of a given low coding level. The underlying synaptic matrix is the result of a generic, slow, long-term synaptic plasticity of two-state synapses, upon repeated presentation of the fixed set of the stimuli to be stored. The neural populations subtending the MF description are classified by the number of stimuli to which their neurons are responsive (multiplicity). This involves 2p + 1 populations for a network storing p memories. The computational complexity of the MF description is then significantly reduced by observing that at low coding levels (f), only a few populations remain relevant: the population of mean multiplicity pf and those of multiplicity of order pf around the mean. The theory is used to produce (predict) bifurcation diagrams (the onset of selective delay activity and the rates in its various stationary states) and to compute the storage capacity of the network (the maximal number of single items used in training for each of which the network can sustain a persistent, selective activity state). This is done in various regions of the space of constitutive parameters for the neurons and for the learning process. The capacity is computed in MF versus potentiation amplitude, ratio of potentiation to depression probability and coding level f. The MF results compare well with recordings of delay activity rate distributions in simulations of the underlying microscopic network of 10,000 neurons.

[1]  N Brunel,et al.  Correlations of cortical Hebbian reverberations: theory versus experiment , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[2]  Daniel J. Amit,et al.  Learning in Neural Networks with Material Synapses , 1994, Neural Computation.

[3]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[4]  Haim Sompolinsky,et al.  STATISTICAL MECHANICS OF NEURAL NETWORKS , 1988 .

[5]  Daniel J. Amit,et al.  Constraints on learning in dynamic synapses , 1992, Network: Computation in Neural Systems.

[6]  Nicolas Brunel,et al.  Dynamics of a recurrent network of spiking neurons before and following learning , 1997 .

[7]  D. Amit,et al.  Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. , 1997, Cerebral cortex.

[8]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[9]  D. Amit,et al.  Retrospective and prospective persistent activity induced by Hebbian learning in a recurrent cortical network , 2003, The European journal of neuroscience.

[10]  W. Senn,et al.  Neocortical pyramidal cells respond as integrate-and-fire neurons to in vivo-like input currents. , 2003, Journal of neurophysiology.

[11]  Paolo Del Giudice,et al.  Long and short-term synaptic plasticity and the formation of working memory: A case study , 2001, Neurocomputing.

[12]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[13]  N Brunel,et al.  Slow stochastic Hebbian learning of classes of stimuli in a recurrent neural network. , 1998, Network.

[14]  N. Brunel,et al.  Firing frequency of leaky intergrate-and-fire neurons with synaptic current dynamics. , 1998, Journal of theoretical biology.

[15]  Daniel J. Amit,et al.  Spike-Driven Synaptic Dynamics Generating Working Memory States , 2003, Neural Computation.

[16]  R. Desimone,et al.  Responses of Macaque Perirhinal Neurons during and after Visual Stimulus Association Learning , 1999, The Journal of Neuroscience.

[17]  Haim Sompolinsky,et al.  The Theory of Neural Networks: The Hebb Rule and Beyond , 1987 .

[18]  I. Morgenstern,et al.  Heidelberg Colloquium on Glassy Dynamics , 1987 .

[19]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[20]  Daniel J. Amit,et al.  Quantitative Study of Attractor Neural Network Retrieving at Low Spike Rates: I , 1991 .

[21]  Daniel J. Amit,et al.  Simulation in neurobiology: theory or experiment? , 1998, Trends in Neurosciences.

[22]  Sompolinsky,et al.  Willshaw model: Associative memory with sparse coding and low firing rates. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[23]  Carlo Fulvi Mari,et al.  Random Networks of Spiking Neurons: Instability in the Xenopus Tadpole Moto-Neural Pattern , 2000, cond-mat/0003263.

[24]  Y. Miyashita,et al.  Neuronal correlate of pictorial short-term memory in the primate temporal cortexYasushi Miyashita , 1988, Nature.

[25]  R. Kühn,et al.  Statistical mechanics for networks of analog neurons , 1990 .

[26]  E. Gardner.  Structure of metastable states in the Hopfield model , 1986 .

[27]  Stefano Fusi.  Spike-driven Synaptic Plasticity for Learning Correlated Patterns of Mean Firing Rates , 2003, Reviews in the neurosciences.

[28]  Nicolas Brunel,et al.  Hebbian Learning of Context in Recurrent Neural Networks , 1996, Neural Computation.

[29]  Ichiro Fujita,et al.  Presumed inhibitory neurons in the macaque inferior temporal cortex: visual response properties and functional interactions with adjacent neurons. , 2004, Journal of neurophysiology.

[30]  D. Amit,et al.  Quantitative study of attractor neural networks retrieving at low spike rates: II. Low-rate retrieval in symmetric networks , 1991 .

[31]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[32]  R. Desimone,et al.  Neural Mechanisms of Visual Working Memory in Prefrontal Cortex of the Macaque , 1996, The Journal of Neuroscience.

[33]  G S Brindley,et al.  Nerve net models of plausible size that perform many simple learning tasks , 1969, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[34]  Stanislas Dehaene,et al.  Networks of Formal Neurons and Memory Palimpsests , 1986 .

[35]  Y. Miyashita,et al.  Neural organization for the long-term memory of paired associates , 1991, Nature.

[36]  N. Brunel.  Persistent activity and the single-cell frequency–current curve in a cortical network model , 2000, Network.

[37]  L. Ricciardi,et al.  Diffusion Processes and Related Topics in Biology. , 1978 .

[38]  Gérard Weisbuch,et al.  Scaling laws for the attractors of Hopfield networks , 1985 .

[39]  Y. Miyashita.  Neuronal correlate of visual associative long-term memory in the primate temporal cortex , 1988, Nature.

[40]  N. B,et al.  Firing Frequency of Leaky Integrate-and-fire Neurons with Synaptic Current Dynamics , 1998 .

[41]  J. Cowan,et al.  Excitatory and inhibitory interactions in localized populations of model neurons. , 1972, Biophysical journal.

[42]  D J Amit,et al.  Multiple-object working memory--a model for behavioral performance. , 2003, Cerebral cortex.

[43]  J. Hammersley,et al.  Diffusion Processes and Related Topics in Biology , 1977 .

[44]  Néstor Parga,et al.  Response of a LIF neuron to inputs filtered with arbitrary time scale , 2004, Neurocomputing.

[45]  S. Amari,et al.  Characteristics of Random Nets of Analog Neuron-Like Elements , 1972, IEEE Trans. Syst. Man Cybern..

[46]  Sandro Romani,et al.  Learning in realistic networks of spiking neurons and spike‐driven plastic synapses , 2005, The European journal of neuroscience.

[47]  J. Buhmann,et al.  Associative memory with high information content. , 1989, Physical review. A, General physics.

[48]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.