Iterative Retrieval and Block Coding in Autoassociative and Heteroassociative Memory

Neural associative memories (NAM) are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Gripon and Berrou (2011) investigated NAM employing block coding, a particular sparse coding method, and reported a significant increase in storage capacity. Here we verify and extend their results for both heteroassociative and recurrent autoassociative networks. For this we provide a new analysis of iterative retrieval in finite autoassociative and heteroassociative networks that allows estimating storage capacity for random and block patterns. Furthermore, we have implemented various retrieval algorithms for block coding and compared them in simulations to our theoretical results and previous simulation data. In good agreement of theory and experiments, we find that finite networks employing block coding can store significantly more memory patterns. However, due to the reduced information per block pattern, it is not possible to significantly increase stored information per synapse. Asymptotically, the information retrieval capacity converges to the known limits C=ln2≈0.69 and C=(ln2)/4≈0.17 also for block coding. We have also implemented very large recurrent networks up to n=2·106 neurons, showing that maximal capacity C≈0.2 bit per synapse occurs for finite networks having a size n≈105 similar to cortical macrocolumns.

[1]  Vincent Gripon,et al.  Associative Memories to Accelerate Approximate Nearest Neighbor Search , 2016, ArXiv.

[2]  P. Dayan,et al.  Optimising synaptic learning rules in linear associative memories , 1991, Biological Cybernetics.

[3]  Friedrich T. Sommer.  Theorie neuronaler Assoziativspeicher: lokales Lernen und iteratives Retrieval von Information , 1994 .

[4]  Friedemann Pulvermüller,et al.  The Neuroscience of Language: On Brain Circuits of Words and Serial Order , 2003 .

[5]  Jim Austin,et al.  Distributed associative memory for use in scene analysis , 1987, Image Vis. Comput..

[6]  J. Albus.  A Theory of Cerebellar Function , 1971 .

[7]  David Willshaw,et al.  Performance characteristics of the associative net , 1992 .

[8]  T. Sejnowski.  Statistical constraints on synaptic plasticity. , 1977, Journal of theoretical biology.

[9]  Günther Palm,et al.  Associative Memory Networks and Sparse Similarity Preserving Codes , 1994 .

[10]  BART KOSKO,et al.  Bidirectional associative memories , 1988, IEEE Trans. Syst. Man Cybern..

[11]  J. Paul Bolam,et al.  Extrinsic Sources of Cholinergic Innervation of the Striatal Complex: A Whole-Brain Mapping Analysis , 2016, Front. Neuroanat..

[12]  B. V. Kryzhanovsky,et al.  A binary pattern classification using potts model , 2008, Optical Memory and Neural Networks.

[13]  Claude Berrou,et al.  Storing Sparse Messages in Networks of Neural Cliques , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[14]  G. Palm,et al.  On associative memory , 2004, Biological Cybernetics.

[15]  Rafal Bogacz,et al.  Model of Familiarity Discrimination in the Perirhinal Cortex , 2004, Journal of Computational Neuroscience.

[16]  Andreas Knoblauch,et al.  Efficient Associative Computation with Discrete Synapses , 2016, Neural Computation.

[17]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[18]  Günther Palm,et al.  Memory Capacities for Synaptic and Structural Plasticity G ¨ Unther Palm , 2022 .

[19]  Vladimir Kryzhanovsky,et al.  Application of Potts-Model Perceptron for Binary Patterns Identification , 2008, ICANN.

[20]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[21]  Kanter.  Potts-glass models of neural networks. , 1988, Physical review. A, General physics.

[22]  Matthias Löwe,et al.  A Comparative Study of Sparse Associative Memories , 2015 .

[23]  E. Gardner,et al.  Maximum Storage Capacity in Neural Networks , 1987 .

[24]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[25]  Anthony V. Robins,et al.  Catastrophic Forgetting and the Pseudorehearsal Solution in Hopfield-type Networks , 1998, Connect. Sci..

[26]  Jehoshua Bruck,et al.  On the number of spurious memories in the Hopfield model , 1990, IEEE Trans. Inf. Theory.

[27]  Friedrich T. Sommer,et al.  Associative Data Storage and Retrieval in Neural Networks , 1996 .

[28]  Victoria J. Hodge,et al.  A Comparison of Standard Spell Checking Algorithms and a Novel Binary Neural Approach , 2003, IEEE Trans. Knowl. Data Eng..

[29]  A. Knoblauch.  Impact of Structural Plasticity on Memory Formation and Decline , 2017 .

[30]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[31]  Andreas Wichert,et al.  Regarding the temporal requirements of a hierarchical Willshaw network , 2012, Neural Networks.

[32]  Vincent Gripon,et al.  Nearly-optimal associative memories based on distributed constant weight codes , 2012, 2012 Information Theory and Applications Workshop.

[33]  J. Knott.  The organization of behavior: A neuropsychological theory , 1951 .

[34]  Andreas Knoblauch,et al.  Neural Associative Memory with Optimal Bayesian Learning , 2011, Neural Computation.

[35]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[36]  Günther Palm,et al.  Iterative retrieval of sparsely coded associative memory patterns , 1996, Neural Networks.

[37]  D. Gabor.  Associative holographic memories , 1969 .

[38]  Vincent Gripon,et al.  A GPU-based associative memory using sparse Neural Networks , 2014, 2014 International Conference on High Performance Computing & Simulation (HPCS).

[39]  Vincent Gripon,et al.  Nearest Neighbour Search using binary neural networks , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[40]  Vincent Gripon,et al.  Sparse Neural Networks With Large Learning Diversity , 2011, IEEE Transactions on Neural Networks.

[41]  Andreas Knoblauch,et al.  Closed-form Expressions for the Moments of the Binomial Probability Distribution , 2008, SIAM J. Appl. Math..

[42]  Günther Palm,et al.  Improved bidirectional retrieval of sparse patterns stored by Hebbian learning , 1999, Neural Networks.

[43]  Andreas Knoblauch,et al.  Neural Associative Memory and the Willshaw--Palm Probability Distribution , 2008, SIAM J. Appl. Math..

[44]  Andreas Knoblauch,et al.  Structural Synaptic Plasticity Has High Memory Capacity and Can Explain Graded Amnesia, Catastrophic Forgetting, and the Spacing Effect , 2014, PloS one.

[45]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[46]  Karl Steinbuch,et al.  Die Lernmatrix , 2004, Kybernetik.

[47]  Andreas Knoblauch,et al.  Zip nets: Efficient associative computation with binary synapses , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[48]  Günther Palm,et al.  Associative memory and threshold control in neural networks , 1987 .

[49]  Rafal Bogacz,et al.  Comparison of computational models of familiarity discrimination in the perirhinal cortex , 2003, Hippocampus.

[50]  Örjan Ekeberg,et al.  A One-Layer Feedback Artificial Neural Network with a Bayesian Learning Rule , 1989, Int. J. Neural Syst..

[51]  Günther Palm,et al.  Bidirectional Retrieval from Associative Memory , 1997, NIPS.

[52]  T. M. Mayhew,et al.  Anatomy of the Cortex: Statistics and Geometry. , 1991 .

[53]  Gilles Laurent,et al.  Olfactory network dynamics and the coding of multidimensional signals , 2002, Nature Reviews Neuroscience.

[54]  Vincent Gripon,et al.  Maximum likelihood associative memories , 2013, 2013 IEEE Information Theory Workshop (ITW).

[55]  F. Y. Wu.  The Potts model , 1982 .

[56]  T. Sejnowski,et al.  Storing covariance with nonlinearly interacting neurons , 1977, Journal of mathematical biology.

[57]  Günther Palm,et al.  Cell assemblies in the cerebral cortex , 2014, Biological Cybernetics.

[58]  Victoria J. Hodge,et al.  A binary neural k-nearest neighbour technique , 2005 .

[59]  Günther Palm,et al.  Information storage and effective data retrieval in sparse matrices , 1989, Neural Networks.

[60]  John Robinson,et al.  Statistical analysis of the dynamics of a sparse associative memory , 1992, Neural Networks.

[61]  F. Sommer,et al.  Structural Plasticity, Effectual Connectivity, and Memory in Cortex , 2016, Front. Neuroanat..

[62]  B.V. Kryzhanovsky,et al.  Vector-neuron models of associative memory , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[63]  G Palm,et al.  Computing with neural networks. , 1987, Science.

[64]  Andreas Knoblauch,et al.  Synchronization and pattern separation in spiking associative memories and visual cortical areas , 2004 .

[65]  Shun-ichi Amari,et al.  Characteristics of sparsely encoded associative memory , 1989, Neural Networks.

[66]  Vincent Gripon,et al.  A study of retrieval algorithms of sparse messages in networks of neural cliques , 2013, ArXiv.

[67]  R. French,et al.  Catastrophic Forgetting in Connectionist Networks: Causes, Consequences and Solutions , 1994 .

[68]  Günther Palm,et al.  Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states , 1992 .

[69]  Andreas Knoblauch.  Optimal synaptic learning in non-linear associative memory , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[70]  Andreas Knoblauch,et al.  Neural associative memory for brain modeling and information retrieval , 2005, Inf. Process. Lett..

[71]  E. Gardner.  The space of interactions in neural network models , 1988 .

[72]  A. Lansner.  Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations , 2009, Trends in Neurosciences.

[73]  E. Rolls.  A theory of hippocampal function in memory , 1996, Hippocampus.

[74]  A. Krikelis,et al.  Associative processing and processors , 1994, Computer.

[75]  Günther Palm,et al.  Neural associative memories and sparse coding , 2013, Neural Networks.

[76]  Andreas Knoblauch.  Optimal Matrix Compression Yields Storage Capacity 1 for Binary Willshaw Associative Memory , 2003, ICANN.

[77]  R. French.  Catastrophic forgetting in connectionist networks , 1999, Trends in Cognitive Sciences.

[78]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .