Mean-Field Approaches to Independent Component Analysis

We develop mean-field approaches for probabilistic independent component analysis (ICA). The sources are estimated from the mean of their posterior distribution and the mixing matrix (and noise level) is estimated by maximum a posteriori (MAP). The latter requires the computation of (a good approximation to) the correlations between sources. For this purpose, we investigate three increasingly advanced mean-field methods: the variational (also known as naive mean field) approach, linear response corrections, and an adaptive version of the Thouless, Anderson and Palmer (1977) (TAP) mean-field approach, which is due to Opper and Winther (2001). The resulting algorithms are tested on a number of problems. On synthetic data, the advanced mean-field approaches are able to recover the correct mixing matrix in cases where the variational mean-field theory fails. For handwritten digits, sparse encoding is achieved using nonnegative source and mixing priors. For speech, the mean-field method is able to separate in the underdetermined (overcomplete) case of two sensors and three sources. One major advantage of the proposed method is its generality and algorithmic simplicity. Finally, we point out several possible extensions of the approaches developed here.

[1]  Huafu Chen,et al.  [Advances in independent component analysis and its application]. , 2003, Sheng wu yi xue gong cheng xue za zhi = Journal of biomedical engineering = Shengwu yixue gongchengxue zazhi.

[2]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[3]  A. R. De Pierro,et al.  On the relation between the ISRA and the EM algorithm for positron emission tomography , 1993, IEEE Trans. Medical Imaging.

[4]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[5]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[6]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..

[7]  Michael I. Jordan,et al.  Mean Field Theory for Sigmoid Belief NetworksMean Field Theory for Sigmoid Belief , 1996 .

[8]  Lars Kai Hansen,et al.  Blind Separation of Noisy Image Mixtures , 2000 .

[9]  Mark A. Girolami,et al.  An Alternative Perspective on Adaptive Independent Component Analysis Algorithms , 1998, Neural Computation.

[10]  Carsten Peterson,et al.  A New Method for Mapping Optimization Problems Onto Neural Networks , 1989, Int. J. Neural Syst..

[11]  Te-Won Lee,et al.  Independent Component Analysis , 1998, Springer US.

[12]  Michael I. Jordan,et al.  Mean Field Theory for Sigmoid Belief Networks , 1996, J. Artif. Intell. Res..

[13]  G. Parisi,et al.  Statistical Field Theory , 1988 .

[14]  M. Opper,et al.  Tractable approximations for probabilistic models: the adaptive Thouless-Anderson-Palmer mean field approach. , 2001, Physical review letters.

[15]  Hagai Attias,et al.  Independent Factor Analysis , 1999, Neural Computation.

[16]  Nikunj C. Oza,et al.  Online Ensemble Learning , 2000, AAAI/IAAI.

[17]  Adel Belouchrani,et al.  Maximum Likelihood Source Separation By the Expectation-Maximization Technique: Deterministic and St , 1995 .

[18]  Kevin H. Knuth.  A Bayesian approach to source separation , 1999 .

[19]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.

[20]  Alle-Jan van der Veen,et al.  Analytical method for blind binary signal separation , 1997, Proceedings of 13th International Conference on Digital Signal Processing.

[21]  Ole Winther,et al.  Gaussian Processes for Classification: Mean-Field Algorithms , 2000, Neural Computation.

[22]  Hilbert J. Kappen,et al.  Efficient Learning in Boltzmann Machines Using Linear Response Theory , 1998, Neural Computation.

[23]  Ole Winther,et al.  Efficient Approaches to Gaussian Process Classification , 1999, NIPS.

[24]  Terrence J. Sejnowski,et al.  Blind source separation of more sources than mixtures using overcomplete representations , 1999, IEEE Signal Processing Letters.

[25]  David J. C. MacKay,et al.  Ensemble Learning for Blind Image Separation and Deconvolution , 2000 .

[26]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[27]  I. S. Gradshteyn,et al.  Table of Integrals, Series, and Products , 1976 .