Overlearning in Marginal Distribution-Based ICA: Analysis and Solutions

The present paper is written as a word of caution, with users of independent component analysis (ICA) in mind, to overlearning phenomena that are often observed.We consider two types of overlearning, typical to high-order statistics based ICA. These algorithms can be seen to maximise the negentropy of the source estimates. The first kind of overlearning results in the generation of spike-like signals, if there are not enough samples in the data or there is a considerable amount of noise present. It is argued that, if the data has power spectrum characterised by 1/f curve, we face a more severe problem, which cannot be solved inside the strict ICA model. This overlearning is better characterised by bumps instead of spikes. Both overlearning types are demonstrated in the case of artificial signals as well as magnetoencephalograms (MEG). Several methods are suggested to circumvent both types, either by making the estimation of the ICA model more robust or by including further modelling of the data.

[1]  D. Turcotte,et al.  Self-organized criticality , 1999 .

[2]  Terrence J. Sejnowski,et al.  Variational Learning of Clusters of Undercomplete Nonsymmetric Independent Components , 2003, J. Mach. Learn. Res..

[3]  R. A. Choudrey.  FLEXIBLE BAYESIAN INDEPENDENT COMPONENT ANALYSIS FOR BLIND SOURCE SEPARATION , 2001 .

[4]  Aapo Hyvärinen,et al.  Complexity Pursuit: Separating Interesting Components from Time Series , 2001, Neural Computation.

[5]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[6]  Motoaki Kawanabe,et al.  A resampling approach to estimate the stability of one-dimensional or multidimensional independent components , 2002, IEEE Transactions on Biomedical Engineering.

[7]  John G. Proakis,et al.  Probability, random variables and stochastic processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[8]  Harri Valpola,et al.  Accurate, Fast and Stable Denoising Source Separation Algorithms , 2004, ICA.

[9]  Jean-Francois Cardoso,et al.  Source separation using higher order moments , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[10]  Andreas Ziehe,et al.  Unmixing Hyperspectral Data , 1999, NIPS.

[11]  Erkki Oja,et al.  Independent Component Analysis for Identification of Artifacts in Magnetoencephalographic Recordings , 1997, NIPS.

[12]  H. Attias.  Independent Component Analysis: ICA, graphical models and variational methods , 2001 .

[13]  P O Hoyer,et al.  Independent component analysis applied to feature extraction from colour and stereo images , 2000, Network.

[14]  Erkki Oja,et al.  Independent component approach to the analysis of EEG and MEG recordings , 2000, IEEE Transactions on Biomedical Engineering.

[15]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[16]  Harri Lappalainen,et al.  Ensemble learning for independent component analysis , 1999 .

[17]  R. Ilmoniemi,et al.  Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain , 1993 .

[18]  Aapo Hyvärinen,et al.  Topographic Independent Component Analysis , 2001, Neural Computation.

[19]  Kari Torkkola,et al.  Blind Separation For Audio Signals - Are We There Yet? , 1999 .

[20]  Tzyy-Ping Jung,et al.  Independent Component Analysis of Electroencephalographic Data , 1995, NIPS.

[21]  Andreas Ziehe,et al.  TDSEP { an e(cid:14)cient algorithm for blind separation using time structure , 1998 .

[22]  A. J. Bell,et al.  INDEPENDENT COMPONENT ANALYSIS OF BIOMEDICAL SIGNALS , 2000 .

[23]  R N Vigário,et al.  Extraction of ocular artefacts from EEG using independent component analysis. , 1997, Electroencephalography and clinical neurophysiology.

[24]  Erkki Oja,et al.  Image Feature Extraction Using Independent Component Analysis , 1996 .

[25]  Barak A. Pearlmutter,et al.  Independent Components of Magnetoencephalography: Localization , 2002, Neural Computation.

[26]  Eric Moulines,et al.  A blind source separation technique using second-order statistics , 1997, IEEE Trans. Signal Process..

[27]  Aapo Hyvärinen,et al.  Sparse Code Shrinkage: Denoising of Nongaussian Data by Maximum Likelihood Estimation , 1999, Neural Computation.

[28]  Matthew J. Beal,et al.  Graphical Models and Variational Methods , 2001 .

[29]  Andreas Ziehe,et al.  Blind Source Separation Techniques for Decomposing Event-Related Brain Signals , 2004, Int. J. Bifurc. Chaos.

[30]  Sidney C. Port,et al.  Probability, Random Variables, and Stochastic Processes—Second Edition (Athanasios Papoulis) , 1986 .

[31]  D. Sornette,et al.  Data-adaptive wavelets and multi-scale singular-spectrum analysis , 1998, chao-dyn/9810034.

[32]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[33]  Pierre Comon.  Independent component analysis - a new concept? signal processing , 1994 .

[34]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[35]  Erkki Oja,et al.  ARTEFACT DETECTION IN ASTROPHYSICAL IMAGE DATA USING INDEPENDENT COMPONENT ANALYSIS , 2001 .

[36]  J. W. Miskin,et al.  Ensemble Learning for Blind Source Separation , 2001 .

[37]  Andrzej Cichocki,et al.  Nonholonomic Orthogonal Learning Algorithms for Blind Source Separation , 2000, Neural Computation.

[38]  C. Jutten.  Calcul neuromimétique et traitement du signal : analyse en composantes indépendantes , 1987 .

[39]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[40]  Aapo Hyvärinen,et al.  Fast and robust fixed-point algorithms for independent component analysis , 1999, IEEE Trans. Neural Networks.

[41]  Aapo Hyvärinen,et al.  New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit , 1997, NIPS.

[42]  Shun-ichi Amari,et al.  Estimating Functions of Independent Component Analysis for Temporally Correlated Signals , 2000, Neural Computation.