Forming sparse representations by local anti-Hebbian learning

How does the brain form a useful representation of its environment? It is shown here that a layer of simple Hebbian units connected by modifiable anti-Hebbian feed-back connections can learn to code a set of patterns in such a way that statistical dependency between the elements of the representation is reduced, while information is preserved. The resulting code is sparse, which is favourable if it is to be used as input to a subsequent supervised associative layer. The operation of the network is demonstrated on two simple problems.

[1]  Michael Satosi Watanabe,et al.  Information-Theoretical Aspects of Inductive and Deductive Inference , 1960, IBM J. Res. Dev..

[2]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[3]  H B Barlow,et al.  Single units and sensation: a neuron doctrine for perceptual psychology? , 1972, Perception.

[4]  Roman Bek,et al.  Discourse on one way in which a quantum-mechanics language on the classical logical base can be built up , 1978, Kybernetika.

[5]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[6]  Satosi Watanabe,et al.  Pattern Recognition: Human and Mechanical , 1985 .

[7]  David Zipser,et al.  Feature Discovery by Competive Learning , 1986, Cogn. Sci..

[8]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[9]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[10]  J. Austin.  Associative memory , 1987 .

[11]  H. Barlow.  Cerebral Cortex as Model Builder , 1987 .

[12]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[13]  P. Foldiak,et al.  Adaptive network for optimal linear feature extraction , 1989, International 1989 Joint Conference on Neural Networks.

[14]  H. B. Barlow,et al.  Finding Minimum Entropy Codes , 1989, Neural Computation.

[15]  Peter Földiák,et al.  Adaptation and decorrelation in the cortex , 1989 .

[16]  Terrence J. Sejnowski,et al.  Unsupervised Learning , 2018, Encyclopedia of GIS.

[17]  S. Grossberg,et al.  Adaptive pattern classification and universal recoding: I. Parallel development and coding of neural feature detectors , 1976, Biological Cybernetics.

[18]  J. Rubner,et al.  Development of feature detectors by self-organization , 2004, Biological Cybernetics.

[19]  C. Malsburg.  Self-organization of orientation sensitive cells in the striate cortex , 2004, Kybernetik.