A modified Armijo rule for the online selection of learning rate of the LMS algorithm

The use of the Armijo rule for the automatic selection of the step size within the class of stochastic gradient descent algorithms is investigated, and the Armijo rule learning rate least mean square (ALR-LMS) algorithm is introduced. This algorithm is derived by integrating an appropriately modified version of the Armijo line search to the least mean square filter update. The analysis of the stability, robustness and the bounds on the parameters which guarantee convergence is conducted, and some practical issues relating the choice of parameters of the ALR-LMS and computational complexity are addressed. Comprehensive simulation results in the system identification and prediction setting support the analysis.

[1]  Paul Tseng,et al.  An Incremental Gradient(-Projection) Method with Momentum Term and Adaptive Stepsize Rule , 1998, SIAM J. Optim..

[2]  B. Farhang-Boroujeny,et al.  Adaptive Filters: Theory and Applications , 1999 .

[3]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[4]  Simon Haykin,et al.  Adaptive filter theory (2nd ed.) , 1991 .

[5]  C.E. Davila,et al.  Line search algorithms for adaptive filtering , 1993, IEEE Trans. Signal Process..

[6]  Javier Calpe-Maravilla,et al.  An easy demonstration of the optimum value of the adaptation constant in the LMS algorithm [FIR filt , 1998 .

[7]  V. John Mathews,et al.  A stochastic gradient adaptive filter with gradient adaptive step size , 1993, IEEE Trans. Signal Process..

[8]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[9]  Jont B. Allen,et al.  Image method for efficiently simulating small‐room acoustics , 1976 .

[10]  Ali H. Sayed,et al.  On the learning mechanism of adaptive filters , 2000, IEEE Trans. Signal Process..

[11]  Behrouz Farhang-Boroujeny,et al.  A new class of gradient adaptive step-size LMS algorithms , 2001, IEEE Trans. Signal Process..

[12]  Neil J. Bershad,et al.  Analysis of the normalized LMS algorithm with Gaussian inputs , 1986, IEEE Trans. Acoust. Speech Signal Process..

[13]  Patrick M. Zurek,et al.  Reducing acoustic feedback in hearing aids , 1995, IEEE Trans. Speech Audio Process..

[14]  N. Bershad,et al.  Time delay estimation using the LMS adaptive filter--Dynamic behavior , 1981 .

[15]  Danilo P. Mandic,et al.  A generalized normalized gradient descent algorithm , 2004, IEEE Signal Processing Letters.

[16]  Pierre Priouret,et al.  Adaptive Algorithms and Stochastic Approximations , 1990, Applications of Mathematics.

[17]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[18]  D. Bertsekas.  On the Goldstein-Levitin-Polyak gradient projection method , 1974, CDC 1974.

[19]  N. Bershad,et al.  The complex LMS adaptive algorithm--Transient weight mean and covariance with applications to the ALE , 1983 .

[20]  Azzedine Zerguine,et al.  LMS-LMF adaptive scheme for echo cancellation , 1996 .

[21]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[22]  Anthony G. Constantinides,et al.  A gradient adaptive step size algorithm for IIR filters , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[23]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[24]  E. Eweda,et al.  Second-order convergence analysis of stochastic adaptive linear filtering , 1983 .

[25]  J. Chambers,et al.  A robust mixed-norm adaptive filter algorithm , 1997, IEEE Signal Processing Letters.