论文信息 - A Hebbian/anti-Hebbian network which optimizes information capacity by orthonormalizing the principal subspace

A Hebbian/anti-Hebbian network which optimizes information capacity by orthonormalizing the principal subspace

A number of recent papers have used the approach of maximising information capacity or mutual information (MI) to examine unsupervised neural networks. In particular, for a linear 'compressing' N-input M-output network (N>M), with noise on the input only, the author (1991) maximised MI when the output represents the principal subspace (or top M principal components) of the input. On the other hand, for a linear 'straight-through' M-input M-output network with noise on the output only, he maximised MI (for a fixed output power) when the outputs are orthonormalised, i.e. decorrelated and of equal variance. A number of algorithms exist to achieve both of these optimal arrangements. In this paper, the author extends this work to develop an algorithm for the case of both input and output noise, with an output power constraint. He finds that it is possible to simplify the obvious algorithm obtained by concatenating the two previous solutions. >

Mark D. Plumbley

[1] Joseph J. Atick,et al. Convergent Algorithm for Sensory Receptive Field Development , 1993, Neural Computation.

[2] Ralph Linsker,et al. Self-organization in a perceptual network , 1988, Computer.

[3] Peter Földiák,et al. Adaptation and decorrelation in the cortex , 1989 .

[4] Mark D. Plumbley. Efficient information transfer and anti-Hebbian neural networks , 1993, Neural Networks.

[5] Joseph J. Atick,et al. Towards a Theory of Early Visual Processing , 1990, Neural Computation.

[6] Ralph Linsker,et al. Local Synaptic Learning Rules Suffice to Maximize Mutual Information in a Linear Network , 1992, Neural Computation.

[7] Kurt Hornik,et al. Convergence analysis of local feature extraction algorithms , 1992, Neural Networks.

[8] Mark D. Plumbley,et al. The effect of receptor signal-to-noise levels on optimal filtering in a sensory system , 1991, [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing.

[9] P. Foldiak,et al. Adaptive network for optimal linear feature extraction , 1989, International 1989 Joint Conference on Neural Networks.

[10] Erkki Oja,et al. Principal components, minor components, and linear neural networks , 1992, Neural Networks.

[11] Terence D. Sanger,et al. Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.