Differential learning algorithms for decorrelation and independent component analysis

Decorrelation and its higher-order generalization, independent component analysis (ICA), are fundamental and important tasks in unsupervised learning, that were studied mainly in the domain of Hebbian learning. In this paper we present a variation of the natural gradient ICA, differential ICA, where the learning relies on the concurrent change of output variables. We interpret the differential learning as the maximum likelihood estimation of parameters with latent variables represented by the random walk model. In such a framework, we derive the differential ICA algorithm and, in addition, we also present the differential decorrelation algorithm that is treated as a special instance of the differential ICA. Algorithm derivation and local stability analysis are given with some numerical experimental results.

[1]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[2]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[3]  Seungjin Choi,et al.  Differential learning and random walk model , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[4]  Seungjin Choi.  Blind Source Separation and Independent Component Analysis : A Review , 2004 .

[5]  Christoph E. Schreiner,et al.  Blind source separation and deconvolution: the dynamic component analysis algorithm , 1998 .

[6]  Marian Stewart Bartlett,et al.  Independent component representations for face recognition , 1998, Electronic Imaging.

[7]  P. Földiák,et al.  Forming sparse representations by local anti-Hebbian learning , 1990, Biological Cybernetics.

[8]  Seungjin Choi,et al.  Factorial Code Representation of Faces for Recognition , 2000, Biologically Motivated Computer Vision.

[9]  Andrzej Cichocki,et al.  Flexible Independent Component Analysis , 2000, J. VLSI Signal Process..

[10]  L Sirovich,et al.  Low-dimensional procedure for the characterization of human faces. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[11]  S. Amari,et al.  Approximate maximum likelihood source separation using the natural gradient , 2001, 2001 IEEE Third Workshop on Signal Processing Advances in Wireless Communications (SPAWC'01). Workshop Proceedings (Cat. No.01EX471).

[12]  Seungjin Choi.  Differential Hebbian-type learning algorithms for decorrelation and independent component analysis , 1998 .

[13]  B. Kosko.  Differential Hebbian learning , 2008 .

[14]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[15]  P. Foldiak,et al.  Adaptive network for optimal linear feature extraction , 1989, International 1989 Joint Conference on Neural Networks.

[16]  Andrzej Cichocki,et al.  Nonholonomic Orthogonal Learning Algorithms for Blind Source Separation , 2000, Neural Computation.

[17]  Seungjin Choi,et al.  Adaptive Differential Decorrelation: A Natural Gradient Algorithm , 2002, ICANN.

[18]  A. J. Bell,et al.  A Unifying Information-Theoretic Framework for Independent Component Analysis , 2000 .

[19]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[20]  Andrzej Cichocki,et al.  Equivariant Nonstationary Source Separation , 2002 .

[21]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[22]  B. Kosco.  Differential Hebbian learning , 1987 .

[23]  S.C. Douglas,et al.  Multichannel blind deconvolution and equalization using the natural gradient , 1997, First IEEE Signal Processing Workshop on Signal Processing Advances in Wireless Communications.

[24]  Kiyotoshi Matsuoka,et al.  A neural net for blind separation of nonstationary signals , 1995, Neural Networks.

[25]  B. Yegnanarayana,et al.  Artificial Neural Networks , 2004 .

[26]  Soo-Young Lee.  Blind Source Separation and Independent Component Analysis: A Review , 2005 .

[27]  Andrzej Cichocki,et al.  Stability Analysis of Learning Algorithms for Blind Source Separation , 1997, Neural Networks.

[28]  Shun-ichi AMARIyy,et al.  NATURAL GRADIENT LEARNING WITH A NONHOLONOMIC CONSTRAINT FOR BLIND DECONVOLUTION OF MULTIPLE CHANNELS , 1999 .

[29]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.