Independent component analysis using Potts models

We explore the extending application of Potts encoding to the task of independent component analysis, which primarily deals with the problem of minimizing the Kullback-Leibler divergence between the joint distribution and the product of all marginal distributions of output components. The competitive mechanism of Potts neurons is used to encode the overlapping projections from observations to output components. Based on these projections, the marginal distributions and the entropy of output components are made tractable for computation and the adaptation of the de-mixing matrix toward independent output components is obtained. The Potts model for ICA is well formulated by an objective function subject to a set of constraints, which leads to a novel energy function. A hybrid of the mean field annealing and the gradient descent method is applied to the energy function. Our approach to independent component analysis presents a new criterion for ICA. The performance of the Potts model for ICA given by our numerical simulations is encouraging.

[1]  Aapo Hyvärinen,et al.  Fast and robust fixed-point algorithms for independent component analysis , 1999, IEEE Trans. Neural Networks.

[2]  Kenneth Rose,et al.  A Deterministic Annealing Approach for Parsimonious Design of Piecewise Regression Models , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Terrence J. Sejnowski,et al.  Edges are the Independent Components of Natural Scenes , 1996, NIPS.

[4]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[5]  Ralph Linsker,et al.  A Local Learning Rule That Enables Information Maximization for Arbitrary Input Distributions , 1997, Neural Computation.

[6]  Rose,et al.  Statistical mechanics and phase transitions in clustering. , 1990, Physical review letters.

[7]  Cheng-Yuan Liou,et al.  Self-organization using Potts models , 1996, Neural Networks.

[8]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[9]  Shun-ichi Amari,et al.  Blind Separation of a Mixture of Uniformly Distributed Source Signals: A Novel Approach , 1999, Neural Computation.

[10]  Erkki Oja,et al.  The nonlinear PCA learning rule in independent component analysis , 1997, Neurocomputing.

[11]  P. Comon,et al.  Contrasts for multichannel blind deconvolution , 1996, IEEE Signal Processing Letters.

[12]  D. Ruderman,et al.  Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[13]  Richard Durbin,et al.  An analogue approach to the travelling salesman problem using an elastic net method , 1987, Nature.

[14]  Jean-Francois Cardoso,et al.  Blind signal separation: statistical principles , 1998, Proc. IEEE.

[15]  Hagai Attias,et al.  Independent Factor Analysis , 1999, Neural Computation.

[16]  A. J. Bell,et al.  A Unifying Information-Theoretic Framework for Independent Component Analysis , 2000 .

[17]  Juan K. Lin,et al.  Faithful Representation of Separable Distributions , 1997, Neural Computation.

[18]  Carsten Peterson,et al.  A New Method for Mapping Optimization Problems Onto Neural Networks , 1989, Int. J. Neural Syst..

[19]  S Makeig,et al.  Analysis of fMRI data by blind separation into independent spatial components , 1998, Human brain mapping.

[20]  Aapo Hyvärinen,et al.  A Fast Fixed-Point Algorithm for Independent Component Analysis , 1997, Neural Computation.

[21]  Jean-Franois Cardoso.  High-Order Contrasts for Independent Component Analysis , 1999, Neural Computation.

[22]  Shun-ichi Amari,et al.  Adaptive Online Learning Algorithms for Blind Separation: Maximum Entropy and Minimum Mutual Information , 1997, Neural Computation.

[23]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[24]  S Makeig,et al.  Blind separation of auditory event-related brain responses into independent components. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[25]  Geoffrey C. Fox,et al.  Constrained Clustering as an Optimization Method , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Magnus Rattray,et al.  Natural gradient descent for on-line learning , 1998 .

[27]  J. H. Hateren,et al.  Independent component filters of natural images compared with simple cells in primary visual cortex , 1998 .

[28]  Erkki Oja,et al.  A class of neural networks for independent component analysis , 1997, IEEE Trans. Neural Networks.

[29]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[30]  Andrzej Cichocki,et al.  Robust techniques for independent component analysis (ICA) with noisy data , 1998, Neurocomputing.