Efficient information transfer and anti-Hebbian neural networks

We attempt to maximise mutual information in a neural network where the output signal is corrupted by noise in the transmission process to the next stage. We assume that the average power available for transmission is limited and that the transmission noise is uncorrelated equal-variance Gaussian noise. Using the Lagrange multiplier technique, we construct a utility function that should be maximised to achieve our optimum. This optimum will be achieved when the output signal components are also uncorrelated and of equal variance. For a linear network with lateral inhibitory connections, using an anti-Hebbian algorithm with modified self-inhibitory connections we guarantee to increase our utility function over time and converge to the required optimum, although not by steepest ascent. For a network with inhibitory interneurons, a combined Hebbian/anti-Hebbian algorithm will similarly increase the utility function over time to find the optimum. This network with interneurons suggests that negative feedback may be used in a perceptual system to optimise forward transmission of information.

[1]  E. Rolls.  The representation and storage of information in neural networks in the primate cerebral cortex and hippocampus , 1989 .

[2]  Richard Durbin,et al.  The computing neuron , 1989 .

[3]  P. Foldiak,et al.  Adaptive network for optimal linear feature extraction , 1989, International 1989 Joint Conference on Neural Networks.

[4]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[5]  Jonathan A. Marshall,et al.  Representation of Uncertainty in Self-Organizing Neural Networks , 1990 .

[6]  Claude E. Shannon,et al.  The mathematical theory of communication , 1950 .

[7]  R. Haber,et al.  Visual Perception , 2018, Encyclopedia of Database Systems.

[8]  J. Dowling.  The Retina: An Approachable Part of the Brain , 1988 .

[9]  Eileen C. Schwab,et al.  Pattern recognition by humans and machines , 1986 .

[10]  S. Grossberg.  The Adaptive Self-Organization of Serial Order in Behavior: Speech, Language, And Motor Control , 1987 .

[11]  Joseph J. Atick,et al.  Convergent Algorithm for Sensory Receptive Field Development , 1993, Neural Computation.

[12]  S. W. Kuffler,et al.  From neuron to brain: A cellular approach to the function of the nervous system , 1976 .

[13]  Mark D. Plumbley,et al.  The effect of receptor signal-to-noise levels on optimal filtering in a sensory system , 1991, [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing.

[14]  Audra E. Kosh,et al.  Linear Algebra and its Applications , 1992 .

[15]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[16]  C.E. Shannon,et al.  Communication in the Presence of Noise , 1949, Proceedings of the IRE.

[17]  Joseph J. Atick,et al.  Towards a Theory of Early Visual Processing , 1990, Neural Computation.

[18]  H. Barlow.  Vision: A theory about the functional role and synaptic mechanism of visual after-effects , 1991 .

[19]  Ernest L. Hall,et al.  Computer Image Processing and Recognition , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Teuvo Kohonen,et al.  Self-organization and associative memory: 3rd edition , 1989 .

[21]  Peter Földiák,et al.  Adaptation and decorrelation in the cortex , 1989 .