Adaptive step size techniques for decorrelation and blind source separation

Careful selection of step size parameters is often necessary to obtain good performance from gradient-based adaptive algorithms for decorrelation and source separation tasks. In this paper, we provide an overview of methods for the on-line calculation of step size parameters for these systems. A particular emphasis is placed on gradient adaptive step sizes for a class of natural gradient algorithms for decorrelation and blind source separation. Simulations verifying their useful behaviors are provided.

[1]  Shun-ichi Amari,et al.  A Theory of Adaptive Pattern Classifiers , 1967, IEEE Trans. Electron. Comput..

[2]  Harold J. Kushner,et al.  wchastic. approximation methods for constrained and unconstrained systems , 1978 .

[3]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[4]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[5]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[6]  Thomas Kailath,et al.  Adaptive algorithms with an automatic gain control feature , 1988 .

[7]  Yann LeCun,et al.  Improving the convergence of back-propagation learning with second-order methods , 1989 .

[8]  S. Karni,et al.  A new convergence factor for adaptive filters , 1989 .

[9]  Sharad Singhal,et al.  Training feed-forward networks with the extended Kalman algorithm , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[10]  Simon Haykin,et al.  Adaptive filter theory (2nd ed.) , 1991 .

[11]  John E. Moody,et al.  Towards Faster Stochastic Gradient Search , 1991, NIPS.

[12]  Dirk T. M. Slock,et al.  On the convergence behavior of the LMS and the normalized LMS algorithms , 1993, IEEE Trans. Signal Process..

[13]  V. John Mathews,et al.  A stochastic gradient adaptive filter with gradient adaptive step size , 1993, IEEE Trans. Signal Process..

[14]  T. Kailath,et al.  A state-space approach to adaptive RLS filtering , 1994, IEEE Signal Processing Magazine.

[15]  H. Kushner,et al.  Analysis of adaptive step size SA algorithms for parameter tracking , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[16]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[17]  Teresa H. Y. Meng,et al.  Normalized data nonlinearities for LMS adaptation , 1994, IEEE Trans. Signal Process..

[18]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[19]  T. Aboulnasr,et al.  A robust variable step size LMS-type algorithm: analysis and simulations , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[20]  Scott C. Douglas.  Generalized gradient adaptive step sizes for stochastic gradient adaptive filters , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[21]  Haim Sompolinsky,et al.  On-line Learning of Dichotomies: Algorithms and Learning Curves. , 1995, NIPS 1995.

[22]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[23]  Shun-ichi Amari,et al.  Self-adaptive neural networks for blind separation of sources , 1996, 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96.

[24]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[25]  Andreas Ziehe,et al.  Adaptive On-line Learning in Changing Environments , 1996, NIPS.

[26]  Shun-ichi Amari,et al.  Blind source separation-semiparametric statistical approach , 1997, IEEE Trans. Signal Process..

[27]  Tyseer Aboulnasr,et al.  A robust variable step-size LMS-type algorithm: analysis and simulations , 1997, IEEE Trans. Signal Process..

[28]  Shun-ichi Amari,et al.  On-line adaptive algorithms in non-stationary environments using a modified conjugate gradient approach , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[29]  Andrzej Cichocki,et al.  Neural networks for blind decorrelation of signals , 1997, IEEE Trans. Signal Process..

[30]  S.C. Douglas,et al.  On-Line Step-Size Selection for Training of Adaptive Systems , 1997, IEEE Signal Processing Magazine.

[31]  Shun-ichi Amari,et al.  Why natural gradient? , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[32]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .