Survey on Independent Component Analysis

A common problem encountered in such disciplines as statistics, data analysis, signal processing, and neural network research, is nding a suitable representation of multivariate data. For computational and conceptual simplicity, such a representation is often sought as a linear transformation of the original data. Well-known linear transformation methods include, for example, principal component analysis, factor analysis, and projection pursuit. A recently developed linear transformation method is independent component analysis (ICA), in which the desired representation is the one that minimizes the statistical dependence of the components of the representation. Such a representation seems to capture the essential structure of the data in many applications. In this paper, we survey the existing theory and methods for ICA.

[1]  M. Bartlett,et al.  A note on the multiplying factors for various chi square approximations , 1954 .

[2]  D. Lawley.  TESTS OF SIGNIFICANCE FOR THE LATENT ROOTS OF COVARIANCE AND CORRELATION MATRICES , 1956 .

[3]  N. L. Johnson,et al.  Multivariate Analysis , 1958, Nature.

[4]  Maurice G. Kendall,et al.  The advanced theory of statistics , 1945 .

[5]  E. B. Andersen,et al.  Modern factor analysis , 1961 .

[6]  H B Barlow,et al.  Single units and sensation: a neuron doctrine for perceptual psychology? , 1972, Perception.

[7]  John W. Tukey,et al.  A Projection Pursuit Algorithm for Exploratory Data Analysis , 1974, IEEE Transactions on Computers.

[8]  Y. Sato,et al.  A Method of Self-Recovering Equalization for Multilevel Amplitude-Modulation Systems , 1975, IEEE Trans. Commun..

[9]  D. Donoho.  ON MINIMUM ENTROPY DECONVOLUTION , 1981 .

[10]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[11]  Yeheskel Bar-ness,et al.  Bootstrapping adaptive interference cancelers - Some practical limitations , 1982 .

[12]  E. Oja,et al.  On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix , 1985 .

[13]  John G. Proakis,et al.  Probability, random variables and stochastic processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[14]  Hong Wang,et al.  Coherent signal-subspace processing for the detection and estimation of angles of arrival of multiple wide-band sources , 1985, IEEE Trans. Acoust. Speech Signal Process..

[15]  Thomas Kailath,et al.  Detection of signals by information theoretic criteria , 1985, IEEE Trans. Acoust. Speech Signal Process..

[16]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[17]  Robin Sibson,et al.  What is projection pursuit , 1987 .

[18]  J. Friedman.  Exploratory Projection Pursuit , 1987 .

[19]  C. Jutten.  Calcul neuromimétique et traitement du signal : analyse en composantes indépendantes , 1987 .

[20]  Erkki Oja,et al.  Neural Networks, Principal Components, and Subspaces , 1989, Int. J. Neural Syst..

[21]  H. B. Barlow,et al.  Finding Minimum Entropy Codes , 1989, Neural Computation.

[22]  Jean-Francois Cardoso,et al.  Source separation using higher order moments , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[23]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  J.G. Daugman,et al.  Entropy reduction and decorrelation in visual coding by oriented neural receptive fields , 1989, IEEE Transactions on Biomedical Engineering.

[25]  R. Liu,et al.  AMUSE: a new blind identification algorithm , 1990, IEEE International Symposium on Circuits and Systems.

[26]  Jean-Francois Cardoso,et al.  Eigen-structure of the fourth-order cumulant tensor with application to the blind source separation problem , 1990, International Conference on Acoustics, Speech, and Signal Processing.

[27]  Lang Tong,et al.  Indeterminacy and identifiability of blind identification , 1991 .

[28]  Jean-Francois Cardoso,et al.  Super-symmetric decomposition of the fourth-order cumulant tensor. Blind identification of more sources than sensors , 1991, [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing.

[29]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[30]  Esfandiar Sorouchyari,et al.  Blind separation of sources, part III: Stability analysis , 1991, Signal Process..

[31]  Dinh Tuan Pham,et al.  Separation of a mixture of independent sources through a maximum likelihood approach , 1992 .

[32]  Joseph J. Atick.  Entropy Minimization: a Design Principle for Sensory Perception? , 1992, Int. J. Neural Syst..

[33]  Jean-Francois Cardoso,et al.  ITERATIVE TECHNIQUES FOR BLIND SOURCE SEPARATION USING ONLY FOURTH-ORDER CUMULANTS , 1992 .

[34]  Ehud Weinstein,et al.  New criteria for blind deconvolution of nonminimum phase systems (channels) , 1990, IEEE Trans. Inf. Theory.

[35]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[36]  Meir Feder,et al.  Multi-channel signal separation by decorrelation , 1993, IEEE Trans. Speech Audio Process..

[37]  Ehud Weinstein,et al.  Super-exponential methods for blind deconvolution , 1993, IEEE Trans. Inf. Theory.

[38]  A. Buja,et al.  Projection Pursuit Indexes Based on Orthonormal Function Expansions , 1993 .

[39]  Jiayang Sun.  Some Practical Aspects of Exploratory Projection Pursuit , 1993, SIAM J. Sci. Comput..

[40]  C. L. Nikias,et al.  Signal processing with higher-order spectra , 1993, IEEE Signal Processing Magazine.

[41]  Eric Moreau,et al.  New self-adaptative algorithms for source separation based on contrast functions , 1993, [1993 Proceedings] IEEE Signal Processing Workshop on Higher-Order Statistics.

[42]  J. Nadal.  Non linear neurons in the low noise limit : a factorial code maximizes information transferJean , 1994 .

[43]  Jean-Francois Cardoso,et al.  Adaptive Source Separation With Uniform Performance , 1994 .

[44]  Juha Karhunen,et al.  Representation and separation of signals using nonlinear PCA type learning , 1994, Neural Networks.

[45]  Schuster,et al.  Separation of a mixture of independent signals using time delayed correlations. , 1994, Physical review letters.

[46]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[47]  Horace Barlow,et al.  What is the computational goal of the neocortex , 1994 .

[48]  Ehud Weinstein,et al.  Criteria for multichannel signal separation , 1994, IEEE Trans. Signal Process..

[49]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[50]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[51]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[52]  Christian Jutten,et al.  Blind source separation for convolutive mixtures , 1995, Signal Process..

[53]  Holger Broman,et al.  On local convergence of a class of blind separation algorithms , 1995, IEEE Trans. Signal Process..

[54]  Gustavo Deco,et al.  Linear redundancy reduction learning , 1995, Neural Networks.

[55]  Jean-Louis Lacoume,et al.  Blind separation of wide-band sources in the frequency domain , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[56]  P. Loubaton,et al.  Adaptive blind separation of convolutive mixtures , 1995 .

[57]  Juha Karhunen,et al.  Generalizations of principal component analysis, optimization problems, and neural networks , 1995, Neural Networks.

[58]  R Hecht-Nielsen,et al.  Replicator neural networks for universal optimal source coding. , 1995, Science.

[59]  Nathalie Delfosse,et al.  Adaptive blind separation of independent sources: A deflation approach , 1995, Signal Process..

[60]  Tzyy-Ping Jung,et al.  Independent Component Analysis of Electroencephalographic Data , 1995, NIPS.

[61]  J. Cardoso,et al.  Maximum Likelihood Source SeparationBy the Expectation-Maximization Technique : Deterministic and Stochastic Implementation , 1995 .

[62]  F. Y. Edgeworth,et al.  The theory of statistics , 1996 .

[63]  Erkki Oja,et al.  Simple Neuron Models for Independent Component Analysis , 1996, Int. J. Neural Syst..

[64]  D. Field,et al.  Natural Image Statistics and Eecient Coding , 1996 .

[65]  Jürgen Schmidhuber,et al.  Semilinear Predictability Minimization Produces Well-Known Feature Detectors , 1996, Neural Computation.

[66]  Barak A. Pearlmutter,et al.  A Context-Sensitive Generalization of ICA , 1996 .

[67]  Juha Karhunen,et al.  A Unified Neural Bigradient Algorithm for robust PCA and MCA , 1996, Int. J. Neural Syst..

[68]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[69]  Pierre Comon,et al.  Independent component analysis, a survey of some algebraic methods , 1996, 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96.

[70]  R. Lambert.  Multichannel blind deconvolution: FIR matrix algebra and separation of multipath mixtures , 1996 .

[71]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[72]  Joos Vandewalle,et al.  A technique for higher-order-only blind source separation , 1996 .

[73]  Andrzej Cichocki,et al.  Robust neural networks with on-line learning for blind identification and blind separation of sources , 1996 .

[74]  Kari Torkkola,et al.  Blind separation of delayed sources based on information maximization , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[75]  Zied Malouche,et al.  Extended anti-Hebbian adaptation for unsupervised source extraction , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[76]  Aapo Hyvärinen,et al.  Purely Logical Neural Principal Component and Independent Component Learning , 1996, ICANN.

[77]  T J Sejnowski,et al.  Learning the higher-order structure of a natural sound. , 1996, Network.

[78]  A. Hyvärinen,et al.  Nonlinear Blind Source Separation by Self-Organizing Maps , 1996 .

[79]  D. Field,et al.  Natural image statistics and efficient coding. , 1996, Network.

[80]  Ehud Weinstein,et al.  Multichannel signal separation: methods and analysis , 1996, IEEE Trans. Signal Process..

[81]  Shun-ichi Amari,et al.  Neural Learning in Structured Parameter Spaces - Natural Riemannian Gradient , 1996, NIPS.

[82]  Aapo Hyvärinen,et al.  New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit , 1997, NIPS.

[83]  Aapo Hyvärinen,et al.  A Fast Fixed-Point Algorithm for Independent Component Analysis , 1997, Neural Computation.

[84]  Erkki Oja,et al.  Independent Component Analysis for Identification of Artifacts in Magnetoencephalographic Recordings , 1997, NIPS.

[85]  Erkki Oja,et al.  The nonlinear PCA learning rule in independent component analysis , 1997, Neurocomputing.

[86]  Erkki Oja,et al.  A class of neural networks for independent component analysis , 1997, IEEE Trans. Neural Networks.

[87]  Juha Karhunen,et al.  A Maximum Likelihood Approach to Nonlinear Blind Source Separation , 1997, ICANN.

[88]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[89]  Andrzej Cichocki,et al.  Modified Herault-Jutten Algorithms for Blind Separation of Sources , 1997, Digit. Signal Process..

[90]  Aapo Hyvrinen.  Independent Component Analysis by Minimization of Mutual Information Independent Component Analysis by Minimization of Mutual Information Independent Component Analysis by Minimization of Mutual Information , 1997 .

[91]  Erkki Oja,et al.  Applications of neural blind separation to signal and image processing , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[92]  Christian Jutten,et al.  Nonlinear source separation: the post-nonlinear mixtures , 1997, ESANN.

[93]  Bruno A. Olshausen,et al.  Inferring Sparse, Overcomplete Image Codes Using an Efficient Coding Framework , 1998, NIPS.

[94]  Eric Moulines,et al.  Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[95]  A. Hyvarinen.  A family of fixed-point algorithms for independent component analysis , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[96]  A. Hyvarinen,et al.  One-unit contrast functions for independent component analysis: a statistical analysis , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[97]  J. Cardoso.  Infomax and maximum likelihood for blind source separation , 1997, IEEE Signal Processing Letters.

[98]  Julia Iiarhunen,et al.  BLIND SOURCE SEPARATION USING LEAST-SQUARES TYPE ADAPTIVE ALGORITHMS , 1997 .

[99]  Te-Won Lee,et al.  Blind source separation of nonlinear mixing models , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[100]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[101]  Colin Fyfe,et al.  An extended exploratory projection pursuit network with linear and nonlinear anti-hebbian lateral connections applied to the cocktail party problem , 1997, Neural Networks.

[102]  Erkki Oja,et al.  Image feature extraction by sparse coding and independent component analysis , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[103]  Petteri Pajunen,et al.  Blind source separation using algorithmic information theory , 1998, Neurocomputing.

[104]  Christopher M. Bishop,et al.  GTM: The Generative Topographic Mapping , 1998, Neural Computation.

[105]  J. H. Hateren,et al.  Independent component filters of natural images compared with simple cells in primary visual cortex , 1998 .

[106]  Jean-Francois Cardoso,et al.  Blind signal separation: statistical principles , 1998, Proc. IEEE.

[107]  Andrzej Cichocki,et al.  Bias removal technique for blind source separation with noisy measurements , 1998 .

[108]  Shun-ichi Amari,et al.  Adaptive blind signal processing-neural network approaches , 1998, Proc. IEEE.

[109]  Andrzej Cichocki,et al.  Robust techniques for independent component analysis (ICA) with noisy data , 1998, Neurocomputing.

[110]  Aapo Hyvärinen,et al.  Independent component analysis in the presence of Gaussian noise by maximizing joint likelihood , 1998, Neurocomputing.

[111]  Jean-François Cardoso,et al.  Multidimensional independent component analysis , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[112]  Erkki Oja,et al.  Independent Component Analysis for Parallel Financial Time Series , 1998, International Conference on Neural Information Processing.

[113]  Erkki Oja,et al.  The nonlinear PCA criterion in blind source separation: Relations with other approaches , 1998, Neurocomputing.

[114]  Erkki Oja,et al.  Independent component analysis by general nonlinear Hebbian-like learning rules , 1998, Signal Process..

[115]  Erkki Oja,et al.  Independent Component Analysis in Wave Decomposition of Auditory Evoked Fields , 1998 .

[116]  Erkki Oja,et al.  An Experimental Comparison of Neural ICA Algorithms , 1998 .

[117]  D. Ruderman,et al.  Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[118]  Aapo Hyvärinen,et al.  Nonlinear independent component analysis: Existence and uniqueness results , 1999, Neural Networks.

[119]  A. J. Bell,et al.  A Unifying Information-Theoretic Framework for Independent Component Analysis , 2000 .

[120]  Erkki Oja,et al.  A fast algorithm for estimating overcomplete ICA bases for image windows , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[121]  Aapo Hyvärinen,et al.  Independent subspace analysis shows emergence of phase and shift invariant features from natural images , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[122]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources , 1999, Neural Computation.

[123]  Aapo Hyvärinen,et al.  Fast and robust fixed-point algorithms for independent component analysis , 1999, IEEE Trans. Neural Networks.

[124]  Terrence J. Sejnowski,et al.  Unsupervised Learning , 2018, Encyclopedia of GIS.

[125]  Aapo Hyvärinen,et al.  Sparse Code Shrinkage: Denoising of Nongaussian Data by Maximum Likelihood Estimation , 1999, Neural Computation.

[126]  Aapo Hyvärinen,et al.  Gaussian moments for noisy independent component analysis , 1999, IEEE Signal Processing Letters.

[127]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.