Local Linear Independent Component Analysis Based on Clustering

In standard Independent Component Analysis (ICA), a linear data model is used for a global description of the data. Even though linear ICA yields meaningful results in many cases, it can provide a crude approximation only for general nonlinear data distributions. In this paper a new structure is proposed, where local ICA models are used in connection with a suitable grouping algorithm clustering the data. The clustering part is responsible for an overall coarse nonlinear representation of the data, while linear ICA models of each cluster are used for describing local features of the data. The goal is to represent the data better than in linear ICA while avoiding computational difficulties related with nonlinear ICA. Several data grouping methods are considered, including standard K-means clustering, self-organizing maps, and neural gas. Connections to existing methods are discussed, and experimental results are given for artificial data and natural images. Furthermore, a general theoretical framework encompassing a large number of methods for representing data is introduced. These range from global, dense representation methods to local, very sparse coding methods. The proposed local ICA methods lie between these two extremes.

[1]  G. C. Tiao,et al.  Bayesian inference in statistical analysis , 1973 .

[2]  Anil K. Jain,et al.  Algorithms for Clustering Data , 1988 .

[3]  Thomas Martinetz,et al.  'Neural-gas' network for vector quantization and its application to time-series prediction , 1993, IEEE Trans. Neural Networks.

[4]  C. Gielen,et al.  Neural computation and self-organizing maps, an introduction , 1993 .

[5]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[6]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[7]  Lucas C. Parra,et al.  Statistical Independence and Novelty Detection with Information Preserving Nonlinear Maps , 1996, Neural Computation.

[8]  Erkki Oja,et al.  A class of neural networks for independent component analysis , 1997, IEEE Trans. Neural Networks.

[9]  Geoffrey E. Hinton,et al.  Modeling the manifolds of images of handwritten digits , 1997, IEEE Trans. Neural Networks.

[10]  Kimmo Valkealahti,et al.  Local Independent Component Analysis by the Self-Organizing Map , 1997, ICANN.

[11]  Bruno A. Olshausen,et al.  Inferring Sparse, Overcomplete Image Codes Using an Efficient Coding Framework , 1998, NIPS.

[12]  Juan K. Lin,et al.  Faithful Representation of Separable Distributions , 1997, Neural Computation.

[13]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[14]  Lei Xu,et al.  Bayesian Ying-Yang machine, clustering and number of clusters , 1997, Pattern Recognit. Lett..

[15]  Nanda Kambhatla,et al.  Dimension Reduction by Local Principal Component Analysis , 1997, Neural Computation.

[16]  Te-Won Lee,et al.  Independent Component Analysis , 1998, Springer US.

[17]  Andrzej Cichocki,et al.  Information-theoretic approach to blind separation of sources in non-linear mixture , 1998, Signal Process..

[18]  Aapo Hyvärinen,et al.  Nonlinear independent component analysis: Existence and uniqueness results , 1999, Neural Networks.

[19]  최승진.  Nonlinear dynamic independent component analysis using state-space and neural network models , 1999 .

[20]  Erkki Oja,et al.  An Experimental Comparison of Neural Algorithms for Independent Component Analysis and Blind Separation , 1999, Int. J. Neural Syst..

[21]  Aapo Hyvärinen,et al.  Fast and robust fixed-point algorithms for independent component analysis , 1999, IEEE Trans. Neural Networks.

[22]  Mark A. Girolami,et al.  Self-Organising Neural Networks: Independent Component Analysis and Blind Source Separation , 1999 .

[23]  Christian Jutten,et al.  Source separation in post-nonlinear mixtures , 1999, IEEE Trans. Signal Process..

[24]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.

[25]  Sejnowski,et al.  ICA MIXTURE MODELS FOR UNSUPERVISED CLASSIFICATION AND AUTOMATIC CONTEXT SWITCHING , 2000 .

[26]  S. Haykin.  Unsupervised adaptive filtering, vol. 1: Blind source separation , 2000 .

[27]  D. Donoho.  NATURE VS . MATH : INTERPRETING INDEPENDENT COMPONENT ANALYSIS IN LIGHT OF COMPUTATIONAL HARMONIC ANALYSIS , 2000 .

[28]  Aapo Hyvärinen,et al.  Emergence of Phase- and Shift-Invariant Features by Decomposition of Natural Images into Independent Feature Subspaces , 2000, Neural Computation.

[29]  P. Hoyer,et al.  TOPOGRAPHIC INDEPENDENT COMPONENT ANALYSIS: VISUALIZING THE DEPENDENCE STRUCTURE , 2001 .