Part 3: Brain science, information science and associative memory model

I review recent progress on the associative memory model, which is a kind of neural network model. First, I introduce this model and a mathematical theory called statistical neurodynamics describing its properties. Next, I discuss an associative memory model with hierarchically correlated memory patterns. Initially, in this model, the state approaches a mixed state that is a superposition of memory patterns. After that, it diverges from the mixed state, and finally converges to a memory pattern. I show that this retrieval dynamics can qualitatively replicate the temporal dynamics of face-responsive neurons in the inferior temporal cortex, which is considered to be the final stage of visual perception in the brain. Finally, I show an unexpected link between associative memory and mobile phones (CDMA). The mathematical structure of the CDMA multi-user detection problem resembles that of the associative memory model. It enables us to apply a theoretical framework of the associative memory model to CDMA.

[1]  Robert O. Winder,et al.  Enumeration of Seven-Argument Threshold Functions , 1965, IEEE Trans. Electron. Comput..

[2]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[3]  N Brunel,et al.  Correlations of cortical Hebbian reverberations: theory versus experiment , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[4]  Nicolas Sourlas,et al.  Spin-glass models as error-correcting codes , 1989, Nature.

[5]  Teuvo Kohonen,et al.  Correlation Matrix Memories , 1972, IEEE Transactions on Computers.

[6]  Sompolinsky,et al.  Statistical mechanics of learning from examples. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[7]  Y. Miyashita,et al.  Neuronal correlate of pictorial short-term memory in the primate temporal cortexYasushi Miyashita , 1988, Nature.

[8]  Masato Okada,et al.  Approximate belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection , 2005, IEEE Transactions on Information Theory.

[9]  James A. Anderson,et al.  A simple neural network generating an interactive memory , 1972 .

[10]  E. Gardner.  The space of interactions in neural network models , 1988 .

[11]  Shun-ichi Amari,et al.  Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements , 1972, IEEE Transactions on Computers.

[12]  T. Ozeki,et al.  Relaxation Dynamics of Decoding Processes of Sourlas Code , 2005 .

[13]  D. Saad,et al.  Statistical mechanics of error-correcting codes , 1999 .

[14]  Y. Kabashima,et al.  Statistical mechanics of lossy data compression using a nonmonotonic perceptron. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[15]  S.-I. Amari,et al.  Neural theory of association and concept-formation , 1977, Biological Cybernetics.

[16]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[17]  Shun-ichi Amari,et al.  Statistical neurodynamics of associative memory , 1988, Neural Networks.

[18]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[19]  T. Watkin,et al.  THE STATISTICAL-MECHANICS OF LEARNING A RULE , 1993 .

[20]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[21]  J Nagumo,et al.  [A model of associative memory]. , 1972, Iyo denshi to seitai kogaku. Japanese journal of medical electronics and biological engineering.

[22]  B. Müller,et al.  The Space of Interactions in Neural Networks , 1990 .

[23]  Sergio Verdu,et al.  Multiuser Detection , 1998 .

[24]  Hansel,et al.  Broken symmetries in multilayered perceptrons. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[25]  S. Yamane,et al.  Population dynamics of face-responsive neurons in the inferior temporal cortex. , 2005, Cerebral cortex.

[26]  Shigeru Yamane,et al.  Neuronal Mechanisms Encoding Global-to-Fine Information in Inferior-Temporal Cortex , 2005, Journal of Computational Neuroscience.

[27]  Y. Miyashita.  Neuronal correlate of visual associative long-term memory in the primate temporal cortex , 1988, Nature.

[28]  Masato Okada,et al.  A hierarchy of macrodynamical equations for associative memory , 1995, Neural Networks.

[29]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[30]  Daniel J. Amit,et al.  Conversion of Temporal Correlations Between Stimuli to Spatial Correlations Between Attractors , 1999, Neural Computation.

[31]  Kenji Kawano,et al.  Global and fine information coded by single neurons in the temporal visual cortex , 1999, Nature.

[32]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[33]  Masato Okada,et al.  One step RSB scheme for the rate distortion function , 2002, cond-mat/0207637.

[34]  E. Gardner,et al.  Zero temperature parallel dynamics for infinite range spin glasses and neural networks , 1987 .

[35]  Masato Okada,et al.  Notions of Associative Memory and Sparse Coding , 1996, Neural Networks.

[36]  Yoshiyuki Kabashima,et al.  Statistical mechanics of typical set decoding. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[37]  Masato Okada,et al.  Generating functional analysis of CDMA detection dynamics , 2005, cond-mat/0507440.

[38]  Kaoru Nakano,et al.  Associatron-A Model of Associative Memory , 1972, IEEE Trans. Syst. Man Cybern..