Analysis of mean-square error and transient speed of the LMS Adaptive algorithm

For the least mean square (LMS) algorithm, we analyze the correlation matrix of the filter coefficient estimation error and the signal estimation error in the transient phase as well as in steady state. We establish the convergence of the second-order statistics as the number of iterations increases, and we derive the exact asymptotic expressions for the mean square errors. In particular, the result for the excess signal estimation error gives conditions under which the LMS algorithm outperforms the Wiener filter with the same number of taps. We also analyze a new measure of transient speed. We do not assume a linear regression model: the desired signal and the data process are allowed to be nonlinearly related. The data is assumed to be an instantaneous transformation of a stationary Markov process satisfying certain ergodic conditions.

[1]  William A. Sethares,et al.  Weak convergence and local stability properties of fixed step size recursive algorithms , 1993, IEEE Trans. Inf. Theory.

[2]  Ehud Weinstein,et al.  Convergence analysis of LMS filters with uncorrelated Gaussian data , 1985, IEEE Trans. Acoust. Speech Signal Process..

[3]  Sean P. Meyn,et al.  Bounds on achievable performance in the identification and adaptive control of time-varying systems , 1999, IEEE Trans. Autom. Control..

[4]  Nurgun Erdol,et al.  Wavelet transform based adaptive filters: analysis and new results , 1996, IEEE Trans. Signal Process..

[5]  B. Anderson,et al.  Performance of adaptive estimation algorithms in dependent random environments , 1980 .

[6]  Carlos S. Kubrusly,et al.  Stochastic approximation algorithms and applications , 1973, CDC 1973.

[7]  Odile Macchi,et al.  Adaptive Processing: The Least Mean Squares Approach with Applications in Transmission , 1995 .

[8]  V. Solo.  The limiting behavior of LMS, " IEEE , 1989 .

[9]  James R. Zeidler,et al.  A performance bound for the LMS estimator , 2000, IEEE Trans. Inf. Theory.

[10]  Onkar Dabeer,et al.  Convergence analysis of the lms and the constant modulus algorithms , 2002 .

[11]  R. C. Bradley.  Basic Properties of Strong Mixing Conditions , 1985 .

[12]  W. Gardner.  Learning characteristics of stochastic-gradient-descent algorithms: A general study, analysis, and critique , 1984 .

[13]  Victor Solo,et al.  The limiting behavior of LMS , 1989, IEEE Trans. Acoust. Speech Signal Process..

[14]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[15]  Victor Solo.  The error variance of LMS with time-varying weights , 1992, IEEE Trans. Signal Process..

[16]  L. Ljung,et al.  Necessary and sufficient conditions for stability of LMS , 1997, IEEE Trans. Autom. Control..

[17]  Tosio Kato.  Perturbation theory for linear operators , 1966 .

[18]  George V. Moustakides,et al.  Locally optimum adaptive signal processing algorithms , 1998, IEEE Trans. Signal Process..

[19]  R. Bitmead.  Convergence in distribution of LMS-type adaptive parameter estimates , 1983 .

[20]  S. Meyn,et al.  Multiplicative ergodicity and large deviations for an irreducible Markov chain , 2000 .

[21]  Lennart Ljung,et al.  Performance analysis of general tracking algorithms , 1995 .

[22]  S. Meyn,et al.  Spectral theory and limit theorems for geometrically ergodic Markov processes , 2002, math/0209200.

[23]  Eweda Eweda,et al.  Convergence analysis of self-adaptive equalizers , 1984, IEEE Trans. Inf. Theory.

[24]  Neil J. Bershad,et al.  On the probability density function of the LMS adaptive filter weights , 1989, IEEE Trans. Acoust. Speech Signal Process..

[25]  Iven M. Y. Mareels,et al.  Quantifying the effects of dimension on the convergence rate of the LMS adaptive FIR estimator , 1998, IEEE Trans. Signal Process..

[26]  E. Eweda,et al.  Second-order convergence analysis of stochastic adaptive linear filtering , 1983 .

[27]  Richard A. Davis,et al.  Time Series: Theory and Methods (2nd ed.). , 1992 .

[28]  James R. Zeidler,et al.  Nonlinear effects in LMS adaptive equalizers , 1999, IEEE Trans. Signal Process..

[29]  Ali H. Sayed,et al.  Error-energy bounds for adaptive gradient algorithms , 1996, IEEE Trans. Signal Process..

[30]  H. Luetkepohl.  The Handbook of Matrices , 1996 .

[31]  G. Moustakides.  EXPONENTIAL CONVERGENCE OF PRODUCTS OF RANDOM MATRICES: APPLICATION TO ADAPTIVE ALGORITHMS , 1998 .

[32]  B. Delyon,et al.  When is adaptive better than optimal? , 1993, IEEE Trans. Autom. Control..

[33]  B. Widrow,et al.  Stationary and nonstationary learning characteristics of the LMS adaptive filter , 1976, Proceedings of the IEEE.

[34]  Pierre Priouret,et al.  Adaptive Algorithms and Stochastic Approximations , 1990, Applications of Mathematics.