Kernel independent component analysis

We present a class of algorithms for independent component analysis (ICA) which use contrast functions based on canonical correlations in a reproducing kernel Hilbert space. On the one hand, we show that our contrast functions are related to mutual information and have desirable mathematical properties as measures of statistical dependence. On the other hand, building on recent developments in kernel methods, we show that these criteria can be computed efficiently. Minimizing these criteria leads to flexible and robust algorithms for ICA. We illustrate with simulations involving a wide variety of source distributions, showing that our algorithms outperform many of the presently known algorithms.

[1]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[2]  Andrei N. Kolmogorov,et al.  On the Shannon theory of information transmission in the case of continuous signals , 1956, IRE Trans. Inf. Theory.

[3]  Solomon Kullback,et al.  Information Theory and Statistics , 1960 .

[4]  H. Widom.  Asymptotic behavior of the eigenvalues of certain integral equations , 1963 .

[5]  H. Widom.  Asymptotic behavior of the eigenvalues of certain integral equations. II , 1964 .

[6]  J. Kettenring,et al.  Canonical Analysis of Several Sets of Variables , 2022 .

[7]  R. Taylor,et al.  The Numerical Treatment of Integral Equations , 1978 .

[8]  Gene H. Golub,et al.  Matrix computations , 1983 .

[9]  J. Friedman,et al.  Estimating Optimal Transformations for Multiple Regression and Correlation. , 1985 .

[10]  Saburou Saitoh,et al.  Theory of Reproducing Kernels and Its Applications , 1988 .

[11]  A. Buja.  Remarks on Functional Canonical Variates, Alternating Least Squares Methods and Ace , 1990 .

[12]  R. Durrett.  Probability: Theory and Examples , 1993 .

[13]  R. Tibshirani,et al.  Generalized Additive Models , 1991 .

[14]  Ronald L. Rivest,et al.  Introduction to Algorithms , 1990 .

[15]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[16]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[17]  P. Bickel.  Efficient and Adaptive Estimation for Semiparametric Models , 1993 .

[18]  B. Silverman,et al.  Canonical correlation analysis when the data are curves. , 1993 .

[19]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[20]  K. Do,et al.  Efficient and Adaptive Estimation for Semiparametric Models. , 1994 .

[21]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[22]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[23]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[24]  D. Chakrabarti,et al.  A fast fixed - point algorithm for independent component analysis , 1997 .

[25]  H. Knutsson,et al.  Learning Canonical Correlations , 1997 .

[26]  Philippe Garat,et al.  Blind separation of mixture of independent sources through a quasi-maximum likelihood approach , 1997, IEEE Trans. Signal Process..

[27]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[28]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[29]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[30]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[31]  Jean-François Cardoso,et al.  Multidimensional independent component analysis , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[32]  Bernhard Schölkopf,et al.  The connection between regularization operators and support vector kernels , 1998, Neural Networks.

[33]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[34]  Stephen J. Wright.  Modified Cholesky Factorizations in Interior-Point Algorithms for Linear Programming , 1999, SIAM J. Optim..

[35]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Sub-Gaussian and Super-Gaussian Sources , 1999, Neural Comput..

[36]  Hagai Attias,et al.  Independent Factor Analysis , 1999, Neural Computation.

[37]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources , 1999, Neural Computation.

[38]  Aapo Hyvärinen,et al.  Fast and robust fixed-point algorithms for independent component analysis , 1999, IEEE Trans. Neural Networks.

[39]  Jean-Franois Cardoso.  High-Order Contrasts for Independent Component Analysis , 1999, Neural Computation.

[40]  Pei Ling Lai,et al.  Ica Using Kernel Canonical Correlation Analysis , 2000 .

[41]  Christopher K. I. Williams,et al.  The Effect of the Input Density Distribution on Kernel-based Classifiers , 2000, ICML.

[42]  B. Schölkopf,et al.  Sparse Greedy Matrix Approximation for Machine Learning , 2000, ICML.

[43]  Christopher K. I. Williams,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[44]  Nello Cristianini,et al.  Classification using String Kernels , 2000 .

[45]  Motoaki Kawanabe,et al.  Kernel Feature Spaces and Nonlinear Blind Souce Separation , 2001, NIPS.

[46]  Katya Scheinberg,et al.  Efficient SVM Training Using Low-Rank Kernel Representations , 2002, J. Mach. Learn. Res..

[47]  Horst Bischof,et al.  Nonlinear Feature Extraction Using Generalized Canonical Correlation Analysis , 2001, ICANN.

[48]  Max Welling,et al.  A Constrained EM Algorithm for Independent Component Analysis , 2001, Neural Computation.

[49]  Nikos A. Vlassis,et al.  Efficient source adaptivity in independent component analysis , 2001, IEEE Trans. Neural Networks.

[50]  E. Oja,et al.  Independent Component Analysis , 2013 .

[51]  V. Roychowdhury,et al.  NON-PARAMETRIC ICA , 2001 .

[52]  Michael I. Jordan,et al.  Tree-dependent Component Analysis , 2002, UAI.

[53]  Nello Cristianini,et al.  Latent Semantic Kernels , 2001, Journal of Intelligent Information Systems.

[54]  Anja Vogler,et al.  An Introduction to Multivariate Statistical Analysis , 2004 .

[55]  Rene F. Swarttouw,et al.  Orthogonal polynomials , 2020, NIST Handbook of Mathematical Functions.