Asymptotic performance analysis of subspace adaptive algorithms introduced in the neural network literature

In the neural network literature, many algorithms have been proposed for estimating the eigenstructure of covariance matrices. We first show that many of these algorithms, when presented in a common framework, show great similitudes with the gradient-like stochastic algorithms usually encountered in the signal processing literature. We derive the asymptotic distribution of these different recursive subspace estimators. A closed-form expression of the covariances in distribution of eigenvectors and associated projection matrix estimators are given and analyzed. In particular, closed-form expressions of the mean square error of these estimators are given. It is found that these covariance matrices have a structure very similar to those describing batch estimation techniques. The accuracy of our asymptotic analysis is checked by numerical simulations, and it is found to be valid not only for a "small" step size but in a very large domain. Finally, convergence speed and deviation from orthonormality of the different algorithms are compared, and several tradeoffs are analyzed.

[1]  Jean Pierre Delmas.  Performances analysis of parameterized adaptive eigenspace algorithms , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[2]  Erkki Oja,et al.  Subspace methods of pattern recognition , 1983 .

[3]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[4]  M. Martone.  Subspace methods for blind identification of multichannel FIR filters using space-time contraction of cumulants , 1998, IEEE Signal Processing Letters.

[5]  E. Oja,et al.  On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix , 1985 .

[6]  T. W. Anderson.  An Introduction to Multivariate Statistical Analysis , 1959 .

[7]  Kurt Hornik,et al.  Convergence of learning algorithms with constant learning rates , 1991, IEEE Trans. Neural Networks.

[8]  Pierre Priouret,et al.  Adaptive Algorithms and Stochastic Approximations , 1990, Applications of Mathematics.

[9]  Nathalie Delfosse,et al.  Adaptive blind separation of independent sources: A deflation approach , 1995, Signal Process..

[10]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[11]  Bin Yang,et al.  Projection approximation subspace tracking , 1995, IEEE Trans. Signal Process..

[12]  E. Oja,et al.  Principal component analysis by homogeneous neural networks, Part I : The weighted subspace criterion , 1992 .

[13]  Bin Yang.  Convergence analysis of the subspace tracking algorithms PAST and PASTd , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[14]  Kurt Hornik,et al.  Convergence analysis of local feature extraction algorithms , 1992, Neural Networks.

[15]  Thierry Chonavel,et al.  Adaptive subspace estimation-application to moving sources localization and blind channel identification , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[16]  Jean Pierre Delmas,et al.  Performance analysis of an adaptive algorithm for tracking dominant subspaces , 1998, IEEE Trans. Signal Process..

[17]  John B. Moore,et al.  Global analysis of Oja's flow for neural networks , 1994, IEEE Trans. Neural Networks.

[18]  Erkki Oja,et al.  Principal components, minor components, and linear neural networks , 1992, Neural Networks.

[19]  G. Golub,et al.  Tracking a few extreme singular values and vectors in signal processing , 1990, Proc. IEEE.

[20]  Bin Yang,et al.  Asymptotic distribution of recursive subspace estimators , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[21]  H. Rutishauser.  Computational aspects of F. L. Bauer's simultaneous iteration method , 1969 .

[22]  Calyampudi Radhakrishna Rao,et al.  Linear Statistical Inference and its Applications , 1967 .

[23]  Erkki Oja,et al.  Principal component analysis , 1998 .

[24]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[25]  Erkki Oja,et al.  Principal component analysis by homogeneous neural networks, part II: Analysis and extentions of the learning algorithm , 1992 .

[26]  Erkki Oja,et al.  Modified Hebbian learning for curve and surface fitting , 1992, Neural Networks.

[27]  D. Brillinger.  Time Series: Data Analysis and Theory. , 1981 .