Neural associative memories and sparse coding

The theoretical, practical and technical development of neural associative memories during the last 40 years is described. The importance of sparse coding of associative memory patterns is pointed out. The use of associative memory networks for large scale brain modeling is also mentioned.

[1]  M. Brecht,et al.  Sparse and powerful cortical spikes , 2010, Current Opinion in Neurobiology.

[2]  Marc'Aurelio Ranzato,et al.  Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition , 2010, ArXiv.

[3]  Udo Seiffert,et al.  Artificial Neural Networks on Massively Parallel Computer Hardware , 2004, ESANN.

[4]  F. Sommer,et al.  Ramsey theory reveals the conditions when sparse coding on subsampled data is unique , 2011 .

[5]  Friedrich T. Sommer,et al.  Adaptive compressed sensing — A new class of self-organizing coding models for neuroscience , 2009, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[6]  W. Krauth,et al.  Storage capacity of memory networks with binary couplings , 1989 .

[7]  Günther Palm,et al.  Cell assemblies in the cerebral cortex , 2014, Biological Cybernetics.

[8]  P. Lennie The Cost of Cortical Computation , 2003, Current Biology.

[9]  Honglak Lee,et al.  Sparse deep belief net model for visual area V2 , 2007, NIPS.

[10]  Eugene M. Izhikevich,et al.  Relating STDP to BCM , 2003, Neural Computation.

[11]  Günther Palm,et al.  Parallel processing for associative and neuronal networks , 1984, Biological Cybernetics.

[12]  Günther Palm,et al.  Scene Segmentation through Synchronization , 2005 .

[13]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[14]  Günther Palm,et al.  Improved bidirectional retrieval of sparse patterns stored by Hebbian learning , 1999, Neural Networks.

[15]  T. Hromádka,et al.  Sparse Representation of Sounds in the Unanesthetized Auditory Cortex , 2008, PLoS biology.

[16]  Y. Dan,et al.  Spike timing-dependent plasticity: a Hebbian learning rule. , 2008, Annual review of neuroscience.

[17]  Eytan Domany,et al.  Models of Neural Networks I , 1991 .

[18]  Ben Goertzel,et al.  A world survey of artificial brain projects, Part I: Large-scale brain simulations , 2010, Neurocomputing.

[19]  Günther Palm,et al.  Memory Capacities for Synaptic and Structural Plasticity G ¨ Unther Palm , 2022 .

[20]  Bruno A. Olshausen,et al.  Learning sparse, overcomplete representations of time-varying natural images , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[21]  Ulrich Rückert,et al.  Digital VLSI Implementation of a Neural Associative Memory , 1997 .

[22]  A. Koulakov,et al.  Sparse Incomplete Representations: A Potential Role of Olfactory Granule Cells , 2011, Neuron.

[23]  Dan Hammerstrom,et al.  Platform performance comparison of PALM network on Pentium 4 and FPGA , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[24]  Gèunther Palm,et al.  Neural Assemblies: An Alternative Approach to Artificial Intelligence , 1982 .

[25]  Shaojuan Zhu,et al.  Simulation of associative neural networks , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[26]  Teuvo Kohonen,et al.  Associative memory. A system-theoretical approach , 1977 .

[27]  H. Horner,et al.  Transients and basins of attraction in neutral network models , 1989 .

[28]  Rudolf Kruse,et al.  of Neural Networks , 2008 .

[29]  Misha Mahowald The Silicon Optic Nerve , 1994 .

[30]  P. Földiák,et al.  Forming sparse representations by local anti-Hebbian learning , 1990, Biological Cybernetics.

[31]  Bernabé Linares-Barranco,et al.  On Real-Time AER 2-D Convolutions Hardware for Neuromorphic Spike-Based Cortical Processing , 2008, IEEE Transactions on Neural Networks.

[32]  Friedrich T. Sommer,et al.  Associative Data Storage and Retrieval in Neural Networks , 1996 .

[33]  A. Hyvärinen,et al.  A multi-layer sparse coding network learns contour coding from natural images , 2002, Vision Research.

[34]  Ulrich Stellmann Ähnlichkeitserhaltende Codierung , 1992 .

[35]  D J Field,et al.  Relations between the statistics of natural images and the response properties of cortical cells. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[36]  D. Johnston,et al.  Regulation of Synaptic Efficacy by Coincidence of Postsynaptic APs and EPSPs , 1997 .

[37]  Günther Palm,et al.  On the Information Storage Capacity of Local Learning Rules , 1992, Neural Computation.

[38]  Günther Palm,et al.  Information storage and effective data retrieval in sparse matrices , 1989, Neural Networks.

[39]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[40]  G. Edelman,et al.  A Universe Of Consciousness: How Matter Becomes Imagination , 2000 .

[41]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[42]  Heik Heinrich Hellmich,et al.  Synaptic plasticity in spiking neural networks (SP2INN): a system approach , 2003, IEEE Trans. Neural Networks.

[43]  H. Markram The Blue Brain Project , 2006, Nature Reviews Neuroscience.

[44]  Professor Dr. Valentino Braitenberg,et al.  On the Texture of Brains , 1977, Heidelberg Science Library.

[45]  H. Sompolinsky,et al.  Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis. , 2012, Annual review of neuroscience.

[46]  Günther Palm,et al.  An Associative Cortical Model of Language Understanding and Action Planning , 2005, IWINAC.

[47]  Jim D. Garside,et al.  SpiNNaker: Design and Implementation of a GALS Multicore System-on-Chip , 2011, JETC.

[48]  Günther Palm,et al.  Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states , 1992 .

[49]  Yann LeCun,et al.  Fast Approximations to Structured Sparse Coding and Applications to Object Classification , 2012, ECCV.

[50]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[51]  W. Gerstner,et al.  Triplets of Spikes in a Model of Spike Timing-Dependent Plasticity , 2006, The Journal of Neuroscience.

[52]  Richard G. Baraniuk,et al.  Sparse Coding via Thresholding and Local Competition in Neural Circuits , 2008, Neural Computation.

[53]  Karl Steinbuch,et al.  Die Lernmatrix , 2004, Kybernetik.

[54]  David Willshaw,et al.  Models of distributed associative memory , 1971 .

[55]  J. A. Anderson,et al.  A memory storage model utilizing spatial correlation functions , 1968, Kybernetik.

[56]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[57]  A. Lansner,et al.  Modelling Hebbian cell assemblies comprised of cortical neurons , 1992 .

[58]  Scott T. Rickard,et al.  Comparing Measures of Sparsity , 2008, IEEE Transactions on Information Theory.

[59]  Ulrich Rückert,et al.  Mixed Mode VLSI Implementation of a Neural Associative Memory , 1999 .

[60]  Günther Palm,et al.  Iterative retrieval of sparsely coded associative memory patterns , 1996, Neural Networks.

[61]  D. Gabor Associative holographic memories , 1969 .

[62]  Peter Földiák,et al.  Sparse coding in the primate cortex , 1998 .

[63]  Y. Dan,et al.  Spike-timing-dependent synaptic plasticity depends on dendritic location , 2005, Nature.

[64]  Wolfgang Kinzel,et al.  Basins of attraction near the critical storage capacity for neural networks with constant stabilities , 1989 .

[65]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[66]  Günther Palm,et al.  Associative Memory Networks and Sparse Similarity Preserving Codes , 1994 .

[67]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[68]  D. Willshaw,et al.  Theories of associative recall , 1970, Quarterly Reviews of Biophysics.

[69]  G. Palm,et al.  On associative memory , 2004, Biological Cybernetics.

[70]  G. Palm Neural Assemblies , 1982, Studies of Brain Function.

[71]  Günther Palm,et al.  LOCAL LEARNING RULES AND SPARSE CODING IN NEURAL NETWORKS , 1990 .

[72]  Patrik O. Hoyer,et al.  Non-negative Matrix Factorization with Sparseness Constraints , 2004, J. Mach. Learn. Res..

[73]  D. Marr A theory of cerebellar cortex , 1969, The Journal of physiology.

[74]  R. Kempter,et al.  Hebbian learning and spiking neurons , 1999 .

[75]  A. Lansner,et al.  A working memory model based on fast Hebbian learning. , 2003, Network.

[76]  Daniel J. Amit,et al.  Modeling brain function: the world of attractor neural networks, 1st Edition , 1989 .

[77]  Kechen Zhang,et al.  A Sparse Object Coding Scheme in Area V4 , 2011, Current Biology.

[78]  Günther Palm,et al.  Combining Visual Attention, Object Recognition and Associative Information Processing in a NeuroBotic System , 2005, Biomimetic Neural Learning for Intelligent Robots.

[79]  Günther Palm,et al.  Does Spike-Timing-Dependent Synaptic Plasticity Couple or Decouple Neurons Firing in Synchrony? , 2012, Front. Comput. Neurosci..

[80]  G Palm,et al.  Computing with neural networks. , 1987, Science.

[81]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[82]  Jean-Pierre Nadal,et al.  Information storage in sparsely coded memory nets , 1990 .

[83]  D. Field,et al.  Natural image statistics and efficient coding. , 1996, Network.

[84]  Martin Rehn,et al.  A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive fields , 2007, Journal of Computational Neuroscience.

[85]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[86]  J. Hawkins,et al.  On Intelligence , 2004 .

[87]  K. D. Punta,et al.  An ultra-sparse code underlies the generation of neural sequences in a songbird , 2002 .

[88]  J. Knott The organization of behavior: A neuropsychological theory , 1951 .

[89]  H Wigström Associative recall and formation of stable modes of activity in neural network models , 1975, Journal of neuroscience research.

[90]  Günther Palm,et al.  Modelling of syntactical processing in the cortex , 2007, Biosyst..

[91]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[92]  E Fransén,et al.  A model of cortical associative memory based on a horizontal network of connected columns. , 1998, Network.

[93]  Stephen A. Ritz,et al.  Distinctive features, categorical perception, and probability learning: some applications of a neural model , 1977 .

[94]  E. Candès,et al.  Sparsity and incoherence in compressive sampling , 2006, math/0611957.

[95]  Andreas Knoblauch,et al.  Neural Associative Memory with Optimal Bayesian Learning , 2011, Neural Computation.

[96]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[97]  Örjan Ekeberg,et al.  Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer , 2008, IBM J. Res. Dev..

[98]  Laurent U. Perrinet,et al.  Role of Homeostasis in Learning Sparse Representations , 2007, Neural Computation.

[99]  Bruno A. Olshausen,et al.  Principles of Image Representation in Visual Cortex , 2003 .

[100]  Martin Rehn,et al.  Storing and restoring visual input with collaborative rank coding and associative memory , 2006, Neurocomputing.

[101]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[102]  Andreas Knoblauch,et al.  Pattern separation and synchronization in spiking associative memories and visual areas , 2001, Neural Networks.

[103]  Günther Palm,et al.  On the asymptotic information storage capacity of neural networks , 1988 .

[104]  Aapo Hyvärinen,et al.  Statistical Models of Natural Images and Cortical Visual Representation , 2010, Top. Cogn. Sci..

[105]  E. Gardner The space of interactions in neural network models , 1988 .

[106]  Shun-ichi Amari,et al.  Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements , 1972, IEEE Transactions on Computers.

[107]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[108]  A. Lansner Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations , 2009, Trends in Neurosciences.

[109]  L Delpuech,et al.  Models of Neural Networks, Deuxième édition, E Domany, JL van Hemmen, K Schulten. Springer Verlag, Marseille (1995) , 1997 .

[110]  A. Krikelis,et al.  Associative processing and processors , 1994, Computer.

[111]  Robert Hecht-Nielsen Confabulation theory - the mechanism of thought , 2007 .

[112]  J L Gallant,et al.  Sparse coding and decorrelation in primary visual cortex during natural vision. , 2000, Science.

[113]  Stephen B. Furber,et al.  Efficient modelling of spiking neural networks on a scalable chip multiprocessor , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[114]  W. Little The existence of persistent states in the brain , 1974 .

[115]  W. Gerstner,et al.  Connectivity reflects coding: a model of voltage-based STDP with homeostasis , 2010, Nature Neuroscience.

[116]  E. Gardner Maximum Storage Capacity in Neural Networks , 1987 .

[117]  W. L. Dunin-Barkowski,et al.  Computer simulation of a cerebellar cortex compartment , 2004, Biological Cybernetics.

[118]  E. B. Baum,et al.  Internal representations for associative memory , 1988, Biological Cybernetics.

[119]  Adel S. Sedra,et al.  Associative memory , 2003 .

[120]  Edmund T. Rolls,et al.  Neuronal selectivity, population sparseness, and ergodicity in the inferior temporal visual cortex , 2007, Biological Cybernetics.

[121]  Misha A. Mahowald,et al.  An Analog VLSI System for Stereoscopic Vision , 1994 .

[122]  Anders Lansner,et al.  Non-commercial Research and Educational Use including without Limitation Use in Instruction at Your Institution, Sending It to Specific Colleagues That You Know, and Providing a Copy to Your Institution's Administrator. All Other Uses, Reproduction and Distribution, including without Limitation Comm , 2022 .