Comparison Between Kanerva's SDM and Hopfield-Type Neural Networks

The Sparse, Distributed Memory (SDM) model (Kanerva, 1984) is compared to Hopfield-type, neural-network models. A mathematical framework for comparing the two models is developed, and the capacity of each model is investigated. The capacity of the SDM can be increased independent of the dimension of the stored vectors, whereas the Hopfield capacity is limited to a fraction of this dimension. The stored information is proportional to the number of connections, and it is shown that this proportionality constant is the same for the SDM, the Hopfield model, and higher-order models. The models are also compared in their ability to store and recall temporal sequences of patterns. The SDM also includes time delays so that contextual information can be used to recover sequences. A generalization of the SDM allows storage of correlated patterns.

[1]  Claude E. Shannon,et al.  The mathematical theory of communication , 1950 .

[2]  D. Marr.  A theory of cerebellar cortex , 1969, The Journal of physiology.

[3]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[4]  Shun-ichi Amari,et al.  Characteristics of randomly connected threshold-element networks and network systems , 1971 .

[5]  J. Albus.  A Theory of Cerebellar Function , 1971 .

[6]  Stephen Grossberg,et al.  Embedding Fields: Underlying Philosophy, Mathematics, and Applications to Psychology, Physiology, and Anatomy , 1971 .

[7]  J Nagumo,et al.  [A model of associative memory]. , 1972, Iyo denshi to seitai kogaku. Japanese journal of medical electronics and biological engineering.

[8]  Kaoru Nakano,et al.  Associatron-A Model of Associative Memory , 1972, IEEE Trans. Syst. Man Cybern..

[9]  Stephen A. Ritz,et al.  Distinctive features, categorical perception, and probability learning: some applications of a neural model , 1977 .

[10]  W. A. Little,et al.  Analytic study of the memory storage capacity of a neural network , 1978 .

[11]  S. Kirkpatrick,et al.  Infinite-ranged models of spin-glasses , 1978 .

[12]  Teuvo Kohonen,et al.  Content-addressable memories , 1980 .

[13]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Stephen Grossberg,et al.  Absolute stability of global pattern formation and parallel memory storage by competitive neural networks , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[15]  Pentti Kanerva,et al.  Self-propagating search: a unified theory of memory (address decoding, cerebellum) , 1984 .

[16]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[17]  伊藤 正男.  The cerebellum and neural control , 1984 .

[18]  Yaser S. Abu-Mostafa,et al.  Information capacity of the Hopfield model , 1985, IEEE Trans. Inf. Theory.

[19]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[20]  David Zipser,et al.  Feature Discovery by Competive Learning , 1986, Cogn. Sci..

[21]  Kanter,et al.  Temporal association in asymmetric neural networks. , 1986, Physical review letters.

[22]  C. L. Giles,et al.  Machine learning using higher order correlation networks , 1986 .

[23]  D Kleinfeld,et al.  Sequential state generation by model neural networks. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[24]  S Grossberg,et al.  Neural dynamics of word recognition and recall: attentional priming, learning, and resonance. , 1986, Psychological review.

[25]  S. Grossberg,et al.  Neural dynamics of word recognition and recall: attentional priming, learning, and resonance. , 1986 .

[26]  Baldi,et al.  Number of stable points for spin-glasses and neural networks of higher orders. , 1987, Physical review letters.

[27]  James D. Keeler,et al.  Basins of attraction of neural network models , 1987 .

[28]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[29]  James D. Keeler,et al.  Capacity for Patterns and Sequences in Kanerva's SDM as Compared to Other Associative Memory Models , 1987, NIPS.

[30]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[31]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[32]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[33]  Philip A. Chou,et al.  The capacity of the Kanerva associative memory , 1989, IEEE Trans. Inf. Theory.

[34]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.