A novel entropy estimator and its application to ICA

We present a new (differential) entropy estimator where the maximum entropy bound is used to approximate the entropy given the observations, and is computed using a numerical procedure. The resulting accurate estimate for the entropy is used to derive a new algorithm to perform independent component analysis (ICA). The new algorithm, ICA by entropy bound minimization (ICA-EBM), adopts a line search procedure, and initially uses updates that constrain the demixing matrix to be orthogonal for robust performance. We present simulation results that demonstrate the superior performance of ICA-EBM and its ability to match sources that come from a wide range of distributions.

[1]  Aapo Hyvärinen,et al.  New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit , 1997, NIPS.

[2]  P. Tichavský,et al.  Efficient variant of algorithm fastica for independent component analysis attaining the cramer-RAO lower bound , 2005, IEEE/SP 13th Workshop on Statistical Signal Processing, 2005.

[3]  Eric Moulines,et al.  A blind source separation technique using second-order statistics , 1997, IEEE Trans. Signal Process..

[4]  Philippe Garat,et al.  Blind separation of mixture of independent sources through a quasi-maximum likelihood approach , 1997, IEEE Trans. Signal Process..

[5]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[6]  J. Cardoso.  On the Performance of Orthogonal Source Separation Algorithms , 1994 .

[7]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[8]  Erkki Oja,et al.  Independent component analysis: algorithms and applications , 2000, Neural Networks.

[9]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[10]  Deniz Erdogmus,et al.  Minimax Mutual Information Approach for Independent Component Analysis , 2004, Neural Computation.

[11]  Ehud Weinstein,et al.  Super-exponential methods for blind deconvolution , 1993, IEEE Trans. Inf. Theory.

[12]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[13]  Jean-François Cardoso.  On the Stability of Source Separation Algorithms , 2000, J. VLSI Signal Process..

[14]  Dinh-Tuan Pham,et al.  Blind separation of instantaneous mixtures of nonstationary sources , 2001, IEEE Trans. Signal Process..

[15]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[16]  Erkki Oja,et al.  Efficient Variant of Algorithm FastICA for Independent Component Analysis Attaining the CramÉr-Rao Lower Bound , 2006, IEEE Transactions on Neural Networks.

[17]  Simone G. O. Fiori,et al.  A Theory for Learning by Weight Flow on Stiefel-Grassman Manifold , 2001, Neural Computation.

[18]  Hualiang Li,et al.  Complex ICA Using Nonlinear Functions , 2008, IEEE Transactions on Signal Processing.

[19]  Pierre Comon,et al.  Comparative Speed Analysis of FastICA , 2007, ICA.

[20]  Xi-Lin Li,et al.  A new gradient search interpretation of super-exponential algorithm , 2006, IEEE Signal Processing Letters.

[21]  Visa Koivunen,et al.  Pearson System based Method for Blind Separation , 2000 .