Information Maximization in Noisy Channels : A Variational Approach

The maximisation of information transmission over noisy channels is a common, albeit generally computationally difficult problem. We approach the difficulty of computing the mutual information for noisy channels by using a variational approximation. We apply the method to several practical examples, including linear compression, population encoding and CDMA. We demonstrate that our approach enables one to calculate encoding and decoding schemes that can be optimised in a principled manner.

[1]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[2]  M. Opper,et al.  Tractable Approximate Belief Propagation , 2001 .

[3]  Matthias Bethge,et al.  Optimal Short-Term Population Coding: When Fisher Information Fails , 2002, Neural Computation.

[4]  Toshiyuki Tanaka.  Analysis of Bit Error Probability of Direct-Sequence CDMA Multiuser Demodulators , 2000, NIPS.

[5]  William M. Campbell,et al.  Mutual Information in Learning Feature Transformations , 2000, ICML.

[6]  Ralph Linsker,et al.  An Application of the Principle of Maximum Information Preservation to Linear Systems , 1988, NIPS.

[7]  Helen Suzanna Becker,et al.  An information-theoretic unsupervised learning algorithm for neural networks , 1993 .

[8]  Nicolas Brunel,et al.  Mutual Information, Fisher Information, and Population Coding , 1998, Neural Computation.

[9]  Martin J. Wainwright,et al.  A new class of upper bounds on the log partition function , 2002, IEEE Transactions on Information Theory.

[10]  J. Nadal,et al.  Nonlinear feedforward networks with stochastic outputs: infomax implies redundancy reduction. , 1998, Network.

[11]  H. B. Barlow,et al.  Unsupervised Learning , 1989, Neural Computation.

[12]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[13]  M. Opper,et al.  Advanced mean field methods: theory and practice , 2001 .

[14]  Sergio Verdú,et al.  Linear multiuser detectors for synchronous code-division multiple-access channels , 1989, IEEE Trans. Inf. Theory.

[15]  Erkki Oja,et al.  Unsupervised learning in neural computation , 2002, Theor. Comput. Sci..

[16]  Gunnar Rätsch,et al.  Kernel PCA and De-Noising in Feature Spaces , 1998, NIPS.

[17]  Ralph Linsker,et al.  Deriving Receptive Fields Using an Optimal Encoding Criterion , 1992, NIPS.

[18]  M. V. Rossum,et al.  In Neural Computation , 2022 .

[19]  T. Jaakkola,et al.  Improving the Mean Field Approximation Via the Use of Mixture Distributions , 1999, Learning in Graphical Models.