Associative Data Storage and Retrieval in Neural Networks

Associative storage and retrieval of binary random patterns in various neural net models with one-step threshold-detection retrieval and local learning rules are the subject of this chapter. For different heteroas-sociation and autoassociation memory tasks specified by the properties of the pattern sets to be stored and upper bounds on the retrieval errors, we compare the performance of various models of finite as well as asymptotically infinite sizes. In infinite models, we consider the case of asymptotically sparse patterns, where the mean activity in a pattern vanishes, and study two asymptotic fidelity requirements: constant error probabilities and vanishing error probabilities. A signal-to-noise ratio analysis is carried out for one retrieval step where the calculations are comparatively straightforward and easy. As performance measures we propose and evaluate information capacities in bits/synapse which also take into account the important property of fault tolerance. For autoassociation we compare one-step and fixed-point retrieval that is analyzed in the literature by methods of statistical mechanics.

[1]  E. Gardner,et al.  Maximum Storage Capacity in Neural Networks , 1987 .

[2]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[3]  Reiner Kree,et al.  in Models of neural networks , 1991 .

[4]  Harry Wechsler,et al.  From Statistics to Neural Networks: Theory and Pattern Recognition Applications , 1996 .

[5]  Isabelle Guyon,et al.  A biologically constrained learning mechanism in networks of formal neurons , 1986 .

[6]  J. Austin.  Associative memory , 1987 .

[7]  S. Kirkpatrick,et al.  Infinite-ranged models of spin-glasses , 1978 .

[8]  P. Peretto,et al.  On learning rules and memory storage abilities of asymmetrical neural networks , 1988 .

[9]  Friedrich T. Sommer.  Theorie neuronaler Assoziativspeicher: lokales Lernen und iteratives Retrieval von Information , 1994 .

[10]  A. R. Gardner-Medwin.  The recall of events through the learning of associations between their parts , 1976, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[11]  Shun-ichi Amari,et al.  Statistical neurodynamics of associative memory , 1988, Neural Networks.

[12]  Günther Palm,et al.  Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states , 1992 .

[13]  Dennis K. Branstad.  Considerations for security in the OSI architecture , 1987, IEEE Network.

[14]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.

[15]  J. Hemmen,et al.  The Hebb rule: storing static and dynamic objects in an associative neural network , 1988 .

[16]  Günther Palm,et al.  Iterative retrieval of sparsely coded associative memory patterns , 1996, Neural Networks.

[17]  Shun-ichi Amari,et al.  Characteristics of randomly connected threshold-element networks and network systems , 1971 .

[18]  L. Personnaz,et al.  Collective computational properties of neural networks: New learning mechanisms. , 1986, Physical review. A, General physics.

[19]  A. Holden.  Models of the stochastic activity of neurones , 1976 .

[20]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[21]  A. M. Uttley,et al.  Conditional Probability Machines and Conditioned Reflexes , 1956 .

[22]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[23]  G Palm,et al.  Computing with neural networks. , 1987, Science.

[24]  John Robinson,et al.  Statistical analysis of the dynamics of a sparse associative memory , 1992, Neural Networks.

[25]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[26]  D. Willshaw,et al.  Theories of associative recall , 1970, Quarterly Reviews of Biophysics.

[27]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[28]  Jean-Pierre Nadal,et al.  Information storage in sparsely coded memory nets , 1990 .

[29]  James A. Anderson,et al.  A simple neural network generating an interactive memory , 1972 .

[30]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[31]  E. Gardner.  The space of interactions in neural network models , 1988 .

[32]  H. Horner,et al.  Transients and basins of attraction in neutral network models , 1989 .

[33]  J. Hemmen.  Nonlinear neural networks near saturation. , 1987 .

[34]  E. Caianiello.  Outline of a theory of thought-processes and thinking machines. , 1961, Journal of theoretical biology.

[35]  José F. Fontanari,et al.  Information processing in synchronous neural networks , 1988 .

[36]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1952, The Journal of physiology.

[37]  G. Palm.  Neural Assemblies , 1982, Studies of Brain Function.

[38]  Günther Palm,et al.  LOCAL LEARNING RULES AND SPARSE CODING IN NEURAL NETWORKS , 1990 .

[39]  Günther Palm,et al.  On the Information Storage Capacity of Local Learning Rules , 1992, Neural Computation.

[40]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[41]  H. Horner.  Neural networks with low levels of activity: Ising vs. McCulloch-Pitts neurons , 1989 .

[42]  J. Buhmann,et al.  Associative memory with high information content. , 1989, Physical review. A, General physics.

[43]  W. Little.  The existence of persistent states in the brain , 1974 .

[44]  Peter Dayan,et al.  Optimal Plasticity from Matrix Memories: What Goes Up Must Come Down , 1990, Neural Computation.

[45]  Professor Moshe Abeles,et al.  Local Cortical Circuits , 1982, Studies of Brain Function.

[46]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.