A Bayesian interpretation for the exponential correlation associative memory

The exponential correlation associative memory (ECAM) is a recurrent neural network model which has large storage capacity and is particularly suited for VLSI hardware implementation. Our aim in this paper is to show how the ECAM model can be entirely derived within a Bayesian framework, thereby providing more insight into the behaviour of this algorithm. The framework for our study is a novel relaxation method which involves direct probabilistic modelling of the pattern corruption mechanism. The parameter of this model is the memoryless probability of error on nodes of the network. This bit-error probability is not only important for the interpretation of the ECAM model, but allows also us to understand some more general properties of Bayesian pattern reconstruction by relaxation. In addition, we demonstrate that both the Hopfield memory and the Boolean network model developed by Aleksander can be regarded as limits of the presented relaxation approach with precise physical meaning in terms of this parameter. To study the dynamical behaviour of our relaxation model, we use the Hamming distance picture of Kanerva which allows us to understand how the bit-error probability evolves during the relaxation process. We also derive a parameter-free expression for the storage capacity of the model which, like a previous result of Chiueh and Goodman, scales exponentially with the number of nodes in the network.

[1]  Steven W. Zucker,et al.  On the Foundations of Relaxation Labeling Processes , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Richard Szeliski,et al.  An Analysis of the Elastic Net Approach to the Traveling Salesman Problem , 1989, Neural Computation.

[3]  Tzi-Dar Chiueh,et al.  Exponential bidirectional associative memories , 1990 .

[4]  Richard Rohwer,et al.  Two Bayesian treatments of the n-tuple recognition method , 1995 .

[5]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[6]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[7]  Philip A. Chou,et al.  The capacity of the Kanerva associative memory , 1989, IEEE Trans. Inf. Theory.

[8]  I. Aleksander.  The logic of connectionist systems , 1989 .

[9]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[10]  Alan L. Yuille,et al.  Generalized Deformable Models, Statistical Physics, and Matching Problems , 1990, Neural Computation.

[11]  C C Wang,et al.  The decision-making properties of discrete multiple exponential bidirectional associative memories , 1995, IEEE Trans. Neural Networks.

[12]  Josef Kittler,et al.  Discrete relaxation , 1990, Pattern Recognit..

[13]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[14]  D. Sherrington,et al.  Theory of associative memory in randomly connected Boolean neural networks , 1989 .

[15]  Igor Aleksander,et al.  Neural computing architectures: the design of brain-like machines , 1989 .

[16]  Philip A. Chou.  The Capacity of the Kanerva Associative Memory is Exponential , 1987, NIPS.

[17]  David Sher,et al.  Improving sampled probability distributions for Markov random fields , 1993, Pattern Recognit. Lett..

[18]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Tzi-Dar Chiueh,et al.  Multivalued associative memories based on recurrent networks , 1993, IEEE Trans. Neural Networks.

[20]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[21]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[22]  Hon-Son Don,et al.  An analysis of high-capacity discrete exponential BAM , 1995, IEEE Trans. Neural Networks.

[23]  Tzi-Dar Chiueh,et al.  VLSI Implementation of a High-Capacity Neural Network Associative Memory , 1989, NIPS.

[24]  Donald Geman,et al.  Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images , 1984 .

[25]  Shlomo Geva,et al.  An Exponential Response Neural Net , 1991, Neural Computation.

[26]  Edwin R. Hancock,et al.  Structural Matching by Discrete Relaxation , 1997, IEEE Trans. Pattern Anal. Mach. Intell..