Flexible and efficient implementations of Bayesian independent component analysis

In this paper we present an empirical Bayes method for flexible and efficient independent component analysis (ICA). The method is flexible with respect to choice of source prior, dimensionality and constraints of the mixing matrix (unconstrained or non-negativity), and structure of the noise covariance matrix. Parameter optimization is handled by variants of the expectation maximization (EM) algorithm: overrelaxed adaptive EM and the easy gradient recipe. These retain the simplicity of EM while converging faster. The required expectations over the source posterior, the sufficient statistics, are estimated with mean field methods: variational and the expectation consistent (EC) framework. We describe the derivation of the EC framework for ICA in detail and give empirical results demonstrating the improved performance. The paper is accompanied by the publicly available Matlab toolbox icaMF.

[1]  Richard M. Everson,et al.  Independent Component Analysis: Principles and Practice , 2001 .

[2]  Ole Winther,et al.  Gaussian Processes for Classification: Mean-Field Algorithms , 2000, Neural Computation.

[3]  Mean Field ICA , 2006 .

[4]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[5]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[6]  Kaare Brandt Petersen,et al.  On the Slow Convergence of EM and VBEM in Low-Noise Linear Models , 2005, Neural Computation.

[7]  Kaare Brandt Petersen,et al.  The EM algorithm in independent component analysis , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[8]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[9]  M. Opper,et al.  Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[10]  Hans Bruun Nielsen,et al.  UCMINF - an Algorithm for Unconstrained, Nonlinear Optimization , 2000 .

[11]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[12]  Ole Winther,et al.  Expectation Consistent Approximate Inference , 2005, J. Mach. Learn. Res..

[13]  G. Orban,et al.  Visual Motion Processing Investigated Using Contrast Agent-Enhanced fMRI in Awake Behaving Monkeys , 2001, Neuron.

[14]  Ole Winther,et al.  Mean-Field Approaches to Independent Component Analysis , 2002, Neural Computation.

[15]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[16]  J. W. Miskin,et al.  Ensemble Learning for Blind Source Separation , 2001 .

[17]  Kaare Brandt Petersen,et al.  State-Space Models: From the EM Algorithm to a Gradient Approach , 2007, Neural Computation.

[18]  Ruslan Salakhutdinov,et al.  Adaptive Overrelaxed Bound Optimization Methods , 2003, ICML.

[19]  Harri Valpola,et al.  Bayesian Ensemble Learning for Nonlinear Factor Analysis , 2000 .

[20]  Mark A. Girolami,et al.  A Variational Method for Learning Sparse and Overcomplete Representations , 2001, Neural Computation.

[21]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine-mediated learning.

[22]  W. Wiegerinck,et al.  Approximate inference techniques with expectation constraints , 2005 .

[23]  Kaare Brandt Petersen,et al.  Bayesian independent component analysis: Variational methods and non-negative decompositions , 2007, Digit. Signal Process..

[24]  Hagai Attias,et al.  A Variational Bayesian Framework for Graphical Models , 1999 .

[25]  Aapo Hyvärinen,et al.  Fast and robust fixed-point algorithms for independent component analysis , 1999, IEEE Trans. Neural Networks.

[26]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[27]  Andrew Zisserman,et al.  Advances in Neural Information Processing Systems (NIPS) , 2007 .

[28]  Jean-Francois Cardoso,et al.  Approximate likelihood for noisy mixtures , 1999 .