Sparse distributed memory and related models

Described here is sparse distributed memory (SDM) as a neural-net associative memory. It is characterized by two weight matrices and by a large internal dimension - the number of hidden units is much larger than the number of input or output units. The first matrix, A, is fixed and possibly random, and the second matrix, C, is modifiable. The SDM is compared and contrasted to (1) computer memory, (2) correlation-matrix memory, (3) feet-forward artificial neural network, (4) cortex of the cerebellum, (5) Marr and Albus models of the cerebellum, and (6) Albus' cerebellar model arithmetic computer (CMAC). Several variations of the basic SDM design are discussed: the selected-coordinate and hyperplane designs of Jaeckel, the pseudorandom associative neural memory of Hassoun, and SDM with real-valued input variables by Prager and Fallside. SDM research conducted mainly at the Research Institute for Advanced Computer Science (RIACS) in 1986-1991 is highlighted.

[1]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[2]  D. Marr.  A theory of cerebellar cortex , 1969, The Journal of physiology.

[3]  J. Albus.  A Theory of Cerebellar Function , 1971 .

[4]  Teuvo Kohonen,et al.  Correlation Matrix Memories , 1972, IEEE Transactions on Computers.

[5]  R. Llinás.  The cortex of the cerebellum. , 1975, Scientific American.

[6]  Teuvo Kohonen,et al.  Content-addressable memories , 1980 .

[7]  G. Miller,et al.  Cognitive science. , 1981, Science.

[8]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[9]  R.M. Dunn,et al.  Brains, behavior, and robotics , 1983, Proceedings of the IEEE.

[10]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[11]  James A. Anderson,et al.  Neurocomputing: Foundations of Research , 1988 .

[12]  David Rogers,et al.  Statistical Prediction with Kanerva's Sparse Distributed Memory , 1988, NIPS.

[13]  M H Hassoun,et al.  Two-Level Neural Network For Deterministic Logic Processing , 1988, Photonics West - Lasers and Applications in Science and Engineering.

[14]  Terrence J. Sejnowski,et al.  NETtalk: a parallel network that learns to read aloud , 1988 .

[15]  Michael J. Flynn,et al.  Sparse, distributed memory prototpe: principles of operation , 1988 .

[16]  J. Keeler.  Comparison Between Kanerva's SDM and Hopfield-Type Neural Networks , 1988, Cogn. Sci..

[17]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[18]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory, Second Edition , 1988, Springer Series in Information Sciences.

[19]  Louis A. Jaeckel.  Two alternate proofs of Wang's lune formula for sparse distributed memory and an integral approximation , 1988 .

[20]  Louis A. Jaeckel.  A class of designs for a sparse distributed memory , 1989 .

[21]  David Rogers,et al.  Predicting Weather Using a Genetic Memory: A Combination of Kanerva's Sparse Distributed Memory with Holland's Genetic Algorithms , 1989, NIPS.

[22]  David Rogers,et al.  Kanerva's Sparse Distributed Memory: an Associative Memory Algorithm Well-Suited to the Connection Machine , 1989, Int. J. High Speed Comput..

[23]  James S. Albus,et al.  The Marr and Albus theories of the cerebellum-two early models of associative memory , 1989, Digest of Papers. COMPCON Spring 89. Thirty-Fourth IEEE Computer Society International Conference: Intellectual Leverage.

[24]  Richard W. Prager,et al.  The modified Kanerva model for automatic speech recognition , 1989 .

[25]  Geoffrey E. Hinton,et al.  Parallel Models of Associative Memory , 1989 .

[26]  Egon E. Loebner,et al.  Intelligent network management and functional cerebellum synthesis , 1989, Digest of Papers. COMPCON Spring 89. Thirty-Fourth IEEE Computer Society International Conference: Intellectual Leverage.

[27]  Pentti Kanerva,et al.  Contour-Map Encoding of Shape for Early Vision , 1989, NIPS.

[28]  Umesh Dwarkanath Joglekar,et al.  Learning to read aloud: A neural network approach using sparse distributed memory , 1989 .

[29]  Michael J. Flynn,et al.  Sparse distributed memory prototype: principles and operation , 1989 .

[30]  Mohamad H. Hassoun,et al.  High Performance Recording Algorithm For Hopfield Model Associative Memories , 1989 .

[31]  Louis A. Jaeckel.  An alternative design for a sparse distributed memory , 1989 .

[32]  Philip A. Chou,et al.  The capacity of the Kanerva associative memory , 1989, IEEE Trans. Inf. Theory.

[33]  David Rogers,et al.  BIRD: A general interface for sparse distributed memory simulators , 1990 .

[34]  Charles C. Jorgensen,et al.  Distributed memory approaches for robotic neural controllers , 1990 .

[35]  Douglas G. Danforth,et al.  An empirical investigation of sparse distributed memory using discrete speech recognition , 1990 .

[36]  Pentti Kanerva,et al.  Efficient packing of patterns in sparse distributed memory by selective weighting of input bits , 1991 .

[37]  Larry M. Manevitz,et al.  Implementing a “Sense of Time” via Entropy in Associative Memories , 1991 .

[38]  James S. Albus,et al.  Outline for a theory of intelligence , 1991, IEEE Trans. Syst. Man Cybern..

[39]  Tomas Nordström,et al.  Designing and using massively parallel computers for artificial neural networks , 1991 .

[40]  Richard W. Prager,et al.  The modified Kanerva model: theory and results for real-time word recognition , 1991 .

[41]  Kimmo Kaski,et al.  Kanerva's sparse distributed memory with multiple hamming thresholds , 1992 .