On Neural Associative Memory Structures: Storage and Retrieval of Sequences in a Chain of Tournaments

Abstract Associative memories enjoy many interesting properties in terms of error correction capabilities, robustness to noise, storage capacity, and retrieval performance, and their usage spans over a large set of applications. In this letter, we investigate and extend tournament-based neural networks, originally proposed by Jiang, Gripon, Berrou, and Rabbat (2016), a novel sequence storage associative memory architecture with high memory efficiency and accurate sequence retrieval. We propose a more general method for learning the sequences, which we call feedback tournament-based neural networks. The retrieval process is also extended to both directions: forward and backward—in other words, any large-enough segment of a sequence can produce the whole sequence. Furthermore, two retrieval algorithms, cache-winner and explore-winner, are introduced to increase the retrieval performance. Through simulation results, we shed light on the strengths and weaknesses of each algorithm.

[1]  Vincent Gripon,et al.  Information, noise, coding, modulation: What about the brain? , 2014, 2014 8th International Symposium on Turbo Codes and Iterative Information Processing (ISTC).

[2]  Gerard J. Rinkus,et al.  A Cortical Sparse Distributed Coding Model Linking Mini- and Macrocolumn-Scale Functionality , 2010, Front. Neuroanat..

[3]  Matthew G. Parker,et al.  Clique-Based Neural Associative Memories with Local Coding and Precoding , 2016, Neural Computation.

[4]  Dileep George,et al.  Sequence memory for prediction, inference and behaviour , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[5]  Vincent Gripon,et al.  Sparse Neural Networks With Large Learning Diversity , 2011, IEEE Transactions on Neural Networks.

[6]  Vincent Gripon,et al.  A Neural Network Model for Solving the Feature Correspondence Problem , 2016, ICANN.

[7]  Vincent Gripon,et al.  Budget Restricted Incremental Learning with Pre-Trained Convolutional Neural Networks and Binary Associative Memories , 2019, J. Signal Process. Syst..

[8]  Vincent Gripon,et al.  Huffman Coding for Storing Non-Uniformly Distributed Messages in Networks of Neural Cliques , 2014, AAAI.

[9]  Bruno A Olshausen,et al.  Sparse coding of sensory inputs , 2004, Current Opinion in Neurobiology.

[10]  Vincent Gripon,et al.  A study of retrieval algorithms of sparse messages in networks of neural cliques , 2013, ArXiv.

[11]  Vincent Gripon,et al.  Nearly-optimal associative memories based on distributed constant weight codes , 2012, 2012 Information Theory and Applications Workshop.

[12]  Vincent Gripon,et al.  Algorithm and Architecture of Fully-Parallel Associative Memories Based on Sparse Clustered Networks , 2014, J. Signal Process. Syst..

[13]  Claude Berrou,et al.  Coded Hopfield networks , 2010, 2010 6th International Symposium on Turbo Codes & Iterative Information Processing.

[14]  John J. Hopfield,et al.  Dense Associative Memory for Pattern Recognition , 2016, NIPS.

[15]  Fabrice Seguin,et al.  A Fully Flexible Circuit Implementation of Clique-Based Neural Networks in 65-nm CMOS , 2019, IEEE Transactions on Circuits and Systems I: Regular Papers.

[16]  Antoine Frappe,et al.  On the Distribution of Clique-Based Neural Networks for Edge AI , 2020, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[17]  Subutai Ahmad,et al.  Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in Neocortex , 2015, Front. Neural Circuits.

[18]  Vincent Gripon,et al.  Algorithm and Architecture for a Low-Power Content-Addressable Memory Based on Sparse Clustered Networks , 2015, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[19]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[20]  Matthew G. Parker,et al.  Neural network associative memories with local coding , 2015, 2015 IEEE 14th Canadian Workshop on Information Theory (CWIT).

[21]  John J. Hopfield,et al.  Searching for Memories, Sudoku, Implicit Check Bits, and the Iterative Use of Not-Always-Correct Rapid Neural Computation , 2006, Neural Computation.

[22]  Jean-Pascal Pfister,et al.  Sequence learning with hidden units in spiking neural networks , 2011, NIPS.

[23]  Matthew G. Parker,et al.  Nested-Clique Network Model of Neural Associative Memory , 2017, Neural Computation.

[24]  Vincent Gripon,et al.  Storing Sequences in Binary Tournament-Based Neural Networks , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Matthias Löwe,et al.  A Comparative Study of Sparse Associative Memories , 2015 .

[26]  Claude Berrou,et al.  Improved Retrieval for Challenging Scenarios in Clique-Based Neural Networks , 2015, IWANN.

[27]  Do-Hyun Kim,et al.  Enhanced storage capacity with errors in scale-free Hopfield neural networks: An analytical study , 2016, PloS one.

[28]  Heiko Hoffmann,et al.  Sparse Associative Memory , 2019, Neural Computation.

[29]  Vincent Gripon,et al.  Finding All Matches in a Database using Binary Neural Networks , 2017 .

[30]  Vincent Gripon,et al.  A GPU-based associative memory using sparse Neural Networks , 2014, 2014 International Conference on High Performance Computing & Simulation (HPCS).

[31]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[32]  Claude Berrou,et al.  Storing Sparse Messages in Networks of Neural Cliques , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[33]  Pierre-Henri Horrein,et al.  Large-Scale Memory of Sequences Using Binary Sparse Neural Networks on GPU , 2017, 2017 International Conference on High Performance Computing & Simulation (HPCS).

[34]  Vincent Gripon,et al.  Algorithm and implementation of an associative memory for oriented edge detection using improved clustered neural networks , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[35]  Aude Billard,et al.  Extended Hopfield Network for Sequence Learning: Application to Gesture Recognition , 2005, ICANN.

[36]  Claude Berrou,et al.  A Connectionist Model of Reading with Error Correction Properties , 2015, LTC.

[37]  Xiaoran Jiang,et al.  Storing Sequences in Binary Neural Networks with High Efficiency. (Mémorisation de séquences dans des réseaux de neurones binaires avec efficacité élevée) , 2014 .