Hebbian learning, its correlation catastrophe, and unlearning

Learning and associative memory are concerned with storing and retrieving activity patterns of a neuronal net. It is considered to be a minimal requirement that the number of patterns that can be stored faithfully is extensive, i.e. is at least proportional to the number of neighbours each neuron interacts with. The main drawback of Hebbian learning, and of any one-shot local learning procedure, is that it cannot store extensively many patterns with activities which vary from pattern to pattern because, being local, it cannot discern global correlations. We critically review the performance of Hebbian unlearning—also proposed as a model of REM sleep—in a network of formal neurons with a distribution of axonal delays. Hebbian unlearning, though as local and unsupervised as Hebbian learning, eliminates undesirable global correlations, handles any spatio-temporal pattern, and improves the network performance greatly—sometimes even saturating a theoretical upper bound. Furthermore, it is shown that unlearning...

[1]  D. Kleinfeld,et al.  "Unlearning" increases the storage capacity of content addressable memories. , 1987, Biophysical journal.

[2]  JA Hobson.  Sleep and dreaming , 1990 .

[3]  J. Winson,et al.  The meaning of dreams. , 1990, Scientific American.

[4]  J. Hobson,et al.  The brain as a dream state generator: an activation-synthesis hypothesis of the dream process. , 1977, The American journal of psychiatry.

[5]  A. Yu. Plakhov,et al.  Neural networks : iterative unlearning algorithm converging to the projector rule matrix , 1994 .

[6]  B. Hars,et al.  Improvement of learning by mesencephalic reticular stimulation during postlearning paradoxical sleep. , 1989, Behavioral and neural biology.

[7]  Thomas H. Brown,et al.  Hebbian Synaptic Plasticity: Evolution of the Contemporary Concept , 1994 .

[8]  J. L. van Hemmen,et al.  Increasing the efficiency of a neural network through unlearning , 1990 .

[9]  J. Leo van Hemmen,et al.  Collective phenomena in neural networks , 1991 .

[10]  E. Gardner.  The space of interactions in neural network models , 1988 .

[11]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[12]  J. Leo van Hemmen,et al.  Statistical Mechanics of Temporal Association in Neural Networks , 1990, NIPS.

[13]  Kanter.  Convergence time in infinite-range neural networks with parallel dynamics at zero temperature. , 1989, Physical review. A, General physics.

[14]  J. J. Hopfield,et al.  ‘Unlearning’ has a stabilizing effect in collective memories , 1983, Nature.

[15]  P. Ross.  Lorenz's Butterfly , 1990 .

[16]  H. Markram,et al.  Regulation of Synaptic Efficacy by Coincidence of Postsynaptic APs and EPSPs , 1997, Science.

[17]  Carlyle Smith,et al.  Paradoxical sleep deprivation applied two days after end of training retards learning , 1988, Physiology & Behavior.

[18]  J. Hemmen,et al.  The Hebb rule: storing static and dynamic objects in an associative neural network , 1988 .

[19]  J. L. van Hemmen,et al.  Unlearning and Its Relevance to REM Sleep: Decorrelating Correlated Data , 1992 .

[20]  Wulfram Gerstner,et al.  Encoding and Decoding of Patterns which are Correlated in Space and Time , 1990, ÖGAI.

[21]  Francis Crick,et al.  The function of dream sleep , 1983, Nature.

[22]  C. Pavlides,et al.  Influences of hippocampal place cell firing in the awake state on the activity of these cells during subsequent sleep episodes , 1989, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[23]  Wulfram Gerstner,et al.  A neuronal learning rule for sub-millisecond temporal coding , 1996, Nature.

[24]  A. Karni,et al.  Dependence on REM sleep of overnight improvement of a perceptual skill. , 1994, Science.

[25]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[26]  J. Leo van Hemmen,et al.  Universality of unlearning , 1994, Neural Networks.