Memory Capacity for Sequences in a Recurrent Network with Biological Constraints

The CA3 region of the hippocampus is a recurrent neural network that is essential for the storage and replay of sequences of patterns that represent behavioral events. Here we present a theoretical framework to calculate a sparsely connected network's capacity to store such sequences. As in CA3, only a limited subset of neurons in the network is active at any one time, pattern retrieval is subject to error, and the resources for plasticity are limited. Our analysis combines an analytical mean field approach, stochastic dynamics, and cellular simulations of a time-discrete McCulloch-Pitts network with binary synapses. To maximize the number of sequences that can be stored in the network, we concurrently optimize the number of active neurons, that is, pattern size, and the firing threshold. We find that for one-step associations (i.e., minimal sequences), the optimal pattern size is inversely proportional to the mean connectivity c, whereas the optimal firing threshold is independent of the connectivity. If the number of synapses per neuron is fixed, the maximum number P of stored sequences in a sufficiently large, nonmodular network is independent of its number N of cells. On the other hand, if the number of synapses scales as the network size to the power of 3/2, the number of sequences P is proportional to N. In other words, sequential memory is scalable. Further-more, we find that there is an optimal ratio r between silent and nonsilent synapses at which the storage capacity = P/[c (1 +r)N] assumes a maximum. For long sequences, the capacity of sequential memory is about one order of magnitude below the capacity for minimal sequences, but otherwise behaves similar to the case of minimal sequences. In a biologically inspired scenario, the information content per synapse is far below theoretical optimality, suggesting that the brain trades off error tolerance against information content in encoding sequential memories.

[1]  E. Gardner,et al.  Maximum Storage Capacity in Neural Networks , 1987 .

[2]  Alessandro Treves,et al.  Stable and Rapid Recurrent Processing in Realistic Autoassociative Memories , 1998, Neural Computation.

[3]  J. Nadal,et al.  Optimal Information Storage and the Distribution of Synaptic Weights Perceptron versus Purkinje Cell , 2004, Neuron.

[4]  Albert K. Lee,et al.  Memory of Sequential Experience in the Hippocampus during Slow Wave Sleep , 2002, Neuron.

[5]  S Dehaene,et al.  Neural networks that learn temporal sequences by selection. , 1987, Proceedings of the National Academy of Sciences of the United States of America.

[6]  Peter Somogyi,et al.  Cell Type and Pathway Dependence of Synaptic AMPA Receptor Number and Variability in the Hippocampus , 1998, Neuron.

[7]  Personnaz,et al.  Storage and retrieval of complex sequences in neural networks. , 1988, Physical review. A, General physics.

[8]  Fiona E. N. LeBeau,et al.  A model of gamma‐frequency network oscillations induced in the rat CA3 region by carbachol in vitro , 2000, The European journal of neuroscience.

[9]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[10]  Adam Prügel-Bennett,et al.  Analysis of synfire chains , 1995 .

[11]  Sompolinsky,et al.  Willshaw model: Associative memory with sparse coding and low firing rates. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[12]  Ad Aertsen,et al.  Stable propagation of synchronous spiking in cortical neural networks , 1999, Nature.

[13]  Joachim M. Buhmann,et al.  Noise-driven temporal association in neural networks , 1987 .

[14]  R. Kempter,et al.  Hebbian learning and spiking neurons , 1999 .

[15]  W. Little.  The existence of persistent states in the brain , 1974 .

[16]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[17]  W B Levy,et al.  A sequence predicting CA3 is a flexible associator that learns and uses context to solve hippocampal‐like tasks , 1996, Hippocampus.

[18]  A I I,et al.  Associative memory : on the ( puzzling ) sparse coding limit , 1990 .

[19]  J. Lisman,et al.  Hippocampal sequence-encoding driven by a cortical multi-item working memory buffer , 2005, Trends in Neurosciences.

[20]  U. Heinemann,et al.  Induction of sharp wave–ripple complexes in vitro and reorganization of hippocampal networks , 2005, Nature Neuroscience.

[21]  J. Isaac,et al.  Evidence for silent synapses: Implications for the expression of LTP , 1995, Neuron.

[22]  G. Buzsáki,et al.  Gamma (40-100 Hz) oscillation in the hippocampus of the behaving rat , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[23]  D. Henze,et al.  Revisiting the role of the hippocampal mossy fiber synapse , 2001, Hippocampus.

[24]  R Huerta,et al.  Explaining synchrony in feed-forward networks: are McCulloch-Pitts neurons good enough? , 2003, Biological cybernetics.

[25]  Jay Buckinghamts.  On setting unit thresholds in an incompletely connected associative net , 1993 .

[26]  B. Barbour,et al.  Properties of Unitary Granule Cell→Purkinje Cell Synapses in Adult Rat Cerebellar Slices , 2002, The Journal of Neuroscience.

[27]  N. S. Barnett,et al.  Private communication , 1969 .

[28]  J. Nadal,et al.  Information capacity of a perceptron , 1992 .

[29]  J. Csicsvari,et al.  Mechanisms of Gamma Oscillations in the Hippocampus of the Behaving Rat , 2003, Neuron.

[30]  A. Draguhn,et al.  Cellular and Network Mechanisms Underlying Spontaneous Sharp Wave–Ripple Complexes in Mouse Hippocampal Slices , 2003, The Journal of physiology.

[31]  Chandan Dasgupta,et al.  A neural network for storing individual patterns in limit cycles , 1991 .

[32]  J. Leo van Hemmen,et al.  Statistical Mechanics of Temporal Association in Neural Networks , 1990, NIPS.

[33]  P. Pavlidis,et al.  Pair Recordings Reveal All-Silent Synaptic Connections and the Postsynaptic Expression of Long-Term Potentiation , 2001, Neuron.

[34]  M. Hasselmo.  Neuromodulation: acetylcholine and memory consolidation , 1999, Trends in Cognitive Sciences.

[35]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[36]  D J Amit,et al.  Neural networks counting chimes. , 1988, Proceedings of the National Academy of Sciences of the United States of America.

[37]  H. Eichenbaum,et al.  Critical role of the hippocampus in memory for sequences of events , 2002, Nature Neuroscience.

[38]  M. Gallagher,et al.  Preserved neuron number in the hippocampus of aged rats with spatial learning deficits. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[39]  Kanter,et al.  Temporal association in asymmetric neural networks. , 1986, Physical review letters.

[40]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[41]  A. Lörincz,et al.  Two‐Phase Computational Model Training Long‐Term Memories in the Entorhinal‐Hippocampal Region , 2000, Annals of the New York Academy of Sciences.

[42]  Menno P. Witter,et al.  Place Cells and Place Recognition Maintained by Direct Entorhinal-Hippocampal Circuitry , 2002, Science.

[43]  城所 良明,et al.  The Salk Institute for Biological Studies(話題) , 1975 .

[44]  Moshe Abeles,et al.  Corticonics: Neural Circuits of Cerebral Cortex , 1991 .

[45]  M Maravall,et al.  Sparsification from dilute connectivity in a neural network model of memory. , 1999, Network.

[46]  William B. Levy,et al.  Temporal Sequence Compression by an Integrate-and-Fire Model of Hippocampal Area CA3 , 2004, Journal of Computational Neuroscience.

[47]  David Willshaw,et al.  Capacity and information efficiency of the associative net , 1997 .

[48]  David Willshaw,et al.  Performance characteristics of the associative net , 1992 .

[49]  R. Traub,et al.  Ripple (approximately 200-Hz) oscillations in temporal structures. , 2000, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[50]  J. Csicsvari,et al.  Replay and Time Compression of Recurring Spike Sequences in the Hippocampus , 1999, The Journal of Neuroscience.

[51]  C. Stevens.  An evolutionary scaling law for the primate visual system and its basis in cortical function , 2001, Nature.

[52]  Paul E. Gilbert,et al.  The role of the hippocampus in memory for the temporal order of a sequence of odors. , 2002, Behavioral neuroscience.

[53]  M. Wilson,et al.  NMDA receptors, place cells and hippocampal spatial memory , 2004, Nature Reviews Neuroscience.

[54]  B. McNaughton,et al.  Reactivation of hippocampal ensemble memories during sleep. , 1994, Science.

[55]  J. Csicsvari,et al.  Ensemble Patterns of Hippocampal CA3-CA1 Neurons during Sharp Wave–Associated Population Events , 2000, Neuron.

[56]  Wulfram Gerstner,et al.  A neuronal learning rule for sub-millisecond temporal coding , 1996, Nature.

[57]  Jean-Pierre Nadal,et al.  Information storage in sparsely coded memory nets , 1990 .

[58]  Gutfreund,et al.  Processing of temporal sequences in neural networks. , 1988, Physical review letters.

[59]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[60]  宁北芳,et al.  疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A , 2005 .