Noisy independent component analysis, maximum likelihood estimation, and competitive learning

We consider the estimation of the data model of independent component analysis when noise is present. We show that the joint maximum likelihood estimation of the independent components and the mixing matrix leads to an objective function already proposed by Olshausen and Field (1996) using a different derivation. Due to the complicated nature of the objective function, we introduce approximations that greatly simplify the optimization problem. We show that the presence of noise implies that the relation between the observed data and the estimates of the independent components is non-linear, and show how to approximate this nonlinearity. In particular, the non-linearity may be approximated by a simple shrinkage operation in the case of super-Gaussian (sparse) data. Using these approximations, we propose an efficient algorithm for approximate maximization of the likelihood. In the case of super-Gaussian components, this may be approximated by simple competitive learning, and in the case of sub-Gaussian components, by anti-competitive learning.

[1]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[2]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[3]  Erkki Oja,et al.  One-unit Learning Rules for Independent Component Analysis , 1996, NIPS.

[4]  Erkki Oja,et al.  Independent component analysis by general nonlinear Hebbian-like learning rules , 1998, Signal Process..

[5]  Philippe Garat,et al.  Blind separation of mixture of independent sources through a quasi-maximum likelihood approach , 1997, IEEE Trans. Signal Process..

[6]  D. Field,et al.  Natural image statistics and efficient coding. , 1996, Network.

[7]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[8]  I. Johnstone,et al.  Wavelet Shrinkage: Asymptopia? , 1995 .

[9]  Erkki Oja,et al.  Applications of neural blind separation to signal and image processing , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[10]  Eric Moulines,et al.  Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[11]  Juha Karhunen,et al.  Blind source separation using least-squares type adaptive algorithms , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[12]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[13]  E. Oja,et al.  Sparse code shrinkage for image denoising , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[14]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.