Notions of Associative Memory and Sparse Coding

This paper summarizes associative memory models and sparse representation of memory in these models. Important properties of the associative memory models are their storage capacity, basin of attraction, and the existence of spurious memories. Sparse coding and nonmonotonic output functions are proposed to improve them. Sparsely coded associative memory model has an extremely large storage capacity which diverges as the mean firing rate of memory patterns approaches 0. The storage capacity strongly depends on the shape of the output function as well as the mean firing rate, even in the case of monotonic output functions. Dynamical properties of the model are analyzed by means of a statistical neurodynamical method. We emphasize the necessity of a feedback mechanism to control the mean firing rate in the recall process. Recently, there have been some experimental results suggesting its existence in the brain. On the other hand, it has been shown that the storage capacity can be markedly improved by replacing the usual monotonic output function with a nonmonotonic one. Another remarkable property of the model with the nonmonotonic neurons is that it seems to have no, or almost no, spurious memory. An associative memory model using nonmonotonic modules with a feedforward inhibition is discussed. The modules consist of two types of threshold units, each of which has a different threshold and can be considered as a biologically plausible representation of the nonmonotonic output function. The above model is compared with the monotonic one. The difference in the storage capacity between the two models becomes small when the sparse patterns are stored. Finally, we discuss the biological plausibility of the discussed associative memory models and sparse coding. Copyright 1996 Elsevier Science Ltd.

[1]  E. Gardner,et al.  Zero temperature parallel dynamics for infinite range spin glasses and neural networks , 1987 .

[2]  S. Amari,et al.  Competition and Cooperation in Neural Nets , 1982 .

[3]  P. Peretto,et al.  On learning rules and memory storage abilities of asymmetrical neural networks , 1988 .

[4]  R. Glauber.  Time‐Dependent Statistics of the Ising Model , 1963 .

[5]  J. Zittartz,et al.  Glauber dynamics of the Little-Hopfield model , 1988 .

[6]  A. E. Patrick,et al.  A probabilistic approach to parallel dynamics for the Little-Hopfield model , 1991 .

[7]  M. Soha,et al.  A distributed approach to lan monitoring using intelligent high performance monitors , 1987, IEEE Network.

[8]  Marius Usher,et al.  Neural Network Modeling of Memory Deterioration in Alzheimer's Disease , 1993, Neural Computation.

[9]  Masahiko Morita,et al.  Associative memory with nonmonotone dynamics , 1993, Neural Networks.

[10]  Shiino,et al.  Large suppression of spurious states in neural networks of nonlinear analog neurons. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[11]  Edmund T. Rolls,et al.  The relative advantages of sparse versus distributed encoding for associative neuronal networks in the brain , 1990 .

[12]  E. Gardner.  The space of interactions in neural network models , 1988 .

[13]  Shun-ichi Amari,et al.  Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements , 1972, IEEE Transactions on Computers.

[14]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[15]  Sompolinsky,et al.  Information storage in neural networks with low levels of activity. , 1987, Physical review. A, General physics.

[16]  Nicolas Brunel,et al.  Dynamics of an attractor neural network converting temporal into spatial correlations Network: Compu , 1994 .

[17]  Daniel J. Amit,et al.  Conversion of Temporal Correlations Between Stimuli to Spatial Correlations Between Attractors , 1999, Neural Computation.

[18]  Edmund T. Rolls,et al.  What determines the capacity of autoassociative memories in the brain? Network , 1991 .

[19]  Boes,et al.  Statistical mechanics for networks of graded-response neurons. , 1991, Physical review. A, Atomic, molecular, and optical physics.

[20]  E. White,et al.  Intrinsic circuitry involving the local axon collaterals of corticothalamic projection cells in mouse SmI cortex , 1987, The Journal of comparative neurology.

[21]  Fumiko Yonezawa,et al.  A new perturbation theory for the dynamics of the Little-Hopfield model , 1995 .

[22]  Kaoru Nakano,et al.  Associatron-A Model of Associative Memory , 1972, IEEE Trans. Syst. Man Cybern..

[23]  Shun-ichi Amari,et al.  Auto-associative memory with two-stage dynamics of nonmonotonic neurons , 1996, IEEE Trans. Neural Networks.

[24]  N Brunel,et al.  Correlations of cortical Hebbian reverberations: theory versus experiment , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[25]  S. Amari,et al.  Sparsely coded associative memories: capacity and dynamical properties , 1991 .

[26]  Kei Kobayashi.  On the capacity of a neuron with a non-monotone output function , 1991 .

[27]  Shun-ichi Amari,et al.  Characteristics of sparsely encoded associative memory , 1989, Neural Networks.

[28]  Fernando J. Pineda,et al.  Dynamics and architecture for neural computation , 1988, J. Complex..

[29]  Masahiko Morita,et al.  Capacity of associative memory using a nonmonotonic neuron model , 1993, Neural Networks.

[30]  E. Gardner,et al.  An Exactly Solvable Asymmetric Neural Network Model , 1987 .

[31]  E T Rolls,et al.  Computational constraints suggest the need for two distinct input systems to the hippocampal CA3 network , 1992, Hippocampus.

[32]  M. Tsodyks,et al.  Capacity of networks with correlated attractors , 1994 .

[33]  Shun-ichi Amari,et al.  Statistical neurodynamics of associative memory , 1988, Neural Networks.

[34]  Opris.  Dynamics of associative memory with a general transfer function. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[35]  R. Erichsen,et al.  Mixture States and Storage with Correlated Patterns in Hopfield'S Model , 1991, Int. J. Neural Syst..

[36]  Kanter,et al.  Associative recall of memory without errors. , 1987, Physical review. A, General physics.

[37]  Hidetoshi Nishimori,et al.  Retrieval process of an associative memory with a general input-output function , 1993, Neural Networks.

[38]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[39]  Waugh,et al.  Associative memory in an analog iterated-map neural network. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[40]  Mohamad H. Hassoun,et al.  Associative neural memories , 1993 .

[41]  M. Shiino,et al.  Replica-symmetric theory of nonlinear analogue neural networks , 1990 .

[42]  D. Amit,et al.  Optimised network for sparsely coded patterns , 1989 .

[43]  Masato Okada,et al.  Robustness to noise of associative memory using non-monotonic analogue neurons , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[44]  Barak A. Pearlmutter.  Learning State Space Trajectories in Recurrent Neural Networks , 1989, Neural Computation.

[45]  Masato Okada,et al.  Statistical Neurodynamics of the Sparsely Encoded Associative Memory , 1996 .

[46]  Hideyuki Koyama,et al.  Analysis of the Recalling Processes of Associative Memory by an Integral Representation of the Sign Function , 1996, Neural Networks.

[47]  Laughton,et al.  Order-parameter flow in symmetric and nonsymmetric fully connected attractor neural networks near saturation. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[48]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[49]  Sompolinsky,et al.  Willshaw model: Associative memory with sparse coding and low firing rates. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[50]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[51]  Y. Frégnac.  The Neural and Molecular Bases of Learning (Life Sciences Report 38 — Dahlem Konferenzen) edited by J. P. Changeux and M. Konishi, John Wiley & Sons, 1987. £55.00 (xiii + 559 pages) ISBN 0 471 91569 6 , 1988, Trends in Neurosciences.

[52]  Gérard Weisbuch,et al.  Scaling laws for the attractors of Hopfield networks , 1985 .

[53]  Masato Okada,et al.  A hierarchy of macrodynamical equations for associative memory , 1995, Neural Networks.

[54]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[55]  Peter Somogyi,et al.  Diverse sources of hippocampal unitary inhibitory postsynaptic potentials and the number of synaptic release sites , 1994, Nature.

[56]  J Nagumo,et al.  [A model of associative memory]. , 1972, Iyo denshi to seitai kogaku. Japanese journal of medical electronics and biological engineering.

[57]  R. Zecchina,et al.  Symmetry breaking in non-monotonic neural networks , 1993 .

[58]  J. Buhmann,et al.  Associative memory with high information content. , 1989, Physical review. A, General physics.

[59]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.

[60]  H. Nishimori,et al.  Retrieval dynamics of associative memory of the Hopfield type , 1993 .

[61]  M. Shiino,et al.  Self-consistent signal-to-noise analysis and its application to analogue neural networks with asymmetric connections , 1992 .

[62]  Teuvo Kohonen,et al.  Representation of Associated Data by Matrix Operators , 1973, IEEE Transactions on Computers.

[63]  N. Tamamaki,et al.  Complete Axon Arborization of a Single CA3 Pyramidal Cell in the Rat Hippocampus, and its Relationship With Postsynaptic Parvalbumin‐containing Interneurons , 1993, The European journal of neuroscience.

[64]  T. Fukai,et al.  Comparative study of spurious-state distribution in analogue neural networks and Boltzmann machine , 1992 .

[65]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[66]  L. F. Cugliandolo,et al.  Correlated Attractors from Uncorrelated Stimuli , 1994, Neural Computation.

[67]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[68]  P. Somogyi,et al.  The hippocampal CA3 network: An in vivo intracellular labeling study , 1994, The Journal of comparative neurology.

[69]  TamGas Geszti,et al.  Physical Models of Neural Networks , 1990 .

[70]  James A. Anderson,et al.  A simple neural network generating an interactive memory , 1972 .

[71]  L. Cooper,et al.  A physiological basis for a theory of synapse modification. , 1987, Science.

[72]  Teuvo Kohonen,et al.  Correlation Matrix Memories , 1972, IEEE Transactions on Computers.

[73]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[74]  Alan M. N. Fu.  Statistical Analysis of an Autoassociative Memory Network , 1994, Neural Computation.

[75]  E. Gardner.  Structure of metastable states in the Hopfield model , 1986 .

[76]  A.C.C. Coolen,et al.  CLOSURE OF MACROSCOPIC LAWS IN DISORDERED SPIN SYSTEMS : A TOY MODEL , 1994 .

[77]  Sompolinsky,et al.  Neural networks with nonlinear synapses and a static noise. , 1986, Physical review. A, General physics.

[78]  Y. Miyashita,et al.  Neuronal correlate of pictorial short-term memory in the primate temporal cortexYasushi Miyashita , 1988, Nature.

[79]  Tomoki Fukai,et al.  Retrieval properties of analog neural networks and the nonmonotonicity of transfer functions , 1995, Neural Networks.

[80]  Shiino,et al.  Self-consistent signal-to-noise analysis of the statistical behavior of analog neural networks and enhancement of the storage capacity. , 1993, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[81]  W. Kinzel.  Learning and pattern recognition in spin glass models , 1985 .

[82]  Masato Okada,et al.  Sparsely encoded associative memory: static synaptic noise and static threshold noise , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[83]  Sherrington,et al.  Dynamics of fully connected attractor neural networks near saturation. , 1993, Physical review letters.

[84]  E. Vaadia,et al.  Firing patterns of single units in the prefrontal cortex and neural network models , 1990 .

[85]  Y. Miyashita.  Neuronal correlate of visual associative long-term memory in the primate temporal cortex , 1988, Nature.

[86]  A. E. Patrick,et al.  On the parallel dynamics for the Little-Hopfield model , 1991 .

[87]  N. Tamamaki,et al.  Hippocampal pyramidal cells excite inhibitory neurons through a single release site , 1993, Nature.

[88]  Hidetoshi Nishimori,et al.  Noise distributions in retrieval dynamics of the Hopfield model , 1994 .

[89]  F. R. Waugh,et al.  Fixed-point attractors in analog neural computation. , 1990, Physical review letters.

[90]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[91]  J. Fontanari,et al.  Information storage and retrieval in synchronous neural networks. , 1987, Physical review. A, General physics.