On gradient adaptation with unit-norm constraints

In this correspondence, we describe gradient-based adaptive algorithms within parameter spaces that are specified by ||w||=1, where ||/spl middot/|| is any vector norm. We provide several algorithm forms and relate them to true gradient procedures via their geometric structures. We also give algorithms that mitigate an inherent numerical instability for L/sub 2/-norm-constrained optimization tasks. Simulations showing the performance of the techniques for independent component analysis are provided.

[1]  U. Helmke,et al.  Optimization and Dynamical Systems , 1994, Proceedings of the IEEE.

[2]  Erkki Oja,et al.  Independent component analysis by general nonlinear Hebbian-like learning rules , 1998, Signal Process..

[3]  Xiang-Sun Zhang,et al.  Neural networks in optimization , 2000 .

[4]  Fa-Long Luo,et al.  A Minor Component Analysis Algorithm , 1997, Neural Networks.

[5]  Dr. M. G. Worster.  Methods of Mathematical Physics , 1947, Nature.

[6]  Sun-Yuan Kung,et al.  Extraction of independent components from hybrid mixture: KuicNet learning algorithm and applications , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[7]  S. Amari,et al.  A self-stabilized minor subspace rule , 1998, IEEE Signal Processing Letters.

[8]  Jenq-Neng Hwang,et al.  Neural networks for intelligent multimedia processing , 1998 .

[9]  Bede Liu,et al.  An iterative algorithm for locating the minimal eigenvector of a symmetric matrix , 1984, ICASSP.

[10]  Norman L. Owsley,et al.  Adaptive data orthogonalization , 1978, ICASSP.

[11]  Qin Lin,et al.  A unified algorithm for principal and minor components extraction , 1998, Neural Networks.

[12]  G. Golub,et al.  Tracking a few extreme singular values and vectors in signal processing , 1990, Proc. IEEE.

[13]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[14]  S. Douglas,et al.  On bias removal and unit norm constraints in equation error adaptive IIR filters , 1996, Conference Record of The Thirtieth Asilomar Conference on Signals, Systems and Computers.

[15]  S. Douglas.  Analysis of an anti-Hebbian adaptive FIR filtering algorithm , 1996 .

[16]  Juha Karhunen,et al.  A Unified Neural Bigradient Algorithm for robust PCA and MCA , 1996, Int. J. Neural Syst..

[17]  K. C. Ho,et al.  Bias removal in equation-error adaptive IIR filters , 1995, IEEE Trans. Signal Process..

[18]  Erkki Oja,et al.  Principal components, minor components, and linear neural networks , 1992, Neural Networks.

[19]  Victor Solo,et al.  Performance analysis of adaptive eigenanalysis algorithms , 1998, IEEE Trans. Signal Process..

[20]  Shun-ichi Amari,et al.  Why natural gradient? , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[21]  S. Amari,et al.  Gradient adaptation under unit-norm constraints , 1998, Ninth IEEE Signal Processing Workshop on Statistical Signal and Array Processing (Cat. No.98TH8381).

[22]  M. Simaan,et al.  IN ThE PRESENCE OF WHITE NOISE , 1985 .

[23]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[24]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[25]  Erkki Oja,et al.  A class of neural networks for independent component analysis , 1997, IEEE Trans. Neural Networks.

[26]  E. Oja,et al.  On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix , 1985 .

[27]  Terrence J. Sejnowski,et al.  Learning Nonlinear Overcomplete Representations for Efficient Coding , 1997, NIPS.

[28]  T. Kailath,et al.  Least squares type algorithm for adaptive implementation of Pisarenko's harmonic retrieval method , 1982 .

[29]  Scott C. Douglas,et al.  Kuicnet Algorithms for Blind Deconvolution , 1998, Neural Networks for Signal Processing VIII. Proceedings of the 1998 IEEE Signal Processing Society Workshop (Cat. No.98TH8378).

[30]  M. Swamy,et al.  A constrained anti-Hebbian learning algorithm for total least-squares estimation with applications to adaptive FIR and IIR filtering , 1994 .

[31]  Andrzej Cichocki,et al.  Neural networks for blind decorrelation of signals , 1997, IEEE Trans. Signal Process..

[32]  S. Amari.  Natural Gradient Works Eciently in Learning , 2022 .