LMS is H∞ Optimal

We show that the celebrated LMS (Least-Mean Squares) adaptive algorithm is H ∞ optimal. In other words, the LMS algorithm, which has long been regarded as an approximate least-mean squares solution, is in fact an exact minimizer of a certain so-called H ∞ error norm. In particular, the LMS minimizes the energy gain from the disturbances to the predicted errors, while the so-called normalized LMS minimizes the energy gain from the disturbances to the filtered errors. Moreover, since these algorithms are central H ∞ filters, they minimize a certain exponential cost function and are thus also risk-sensitive optimal (in the sense of Whittle). We discuss the various implications of these results, and show how they provide theoretical justification for the widely observed excellent robustness properties of the LMS filter.

[1]  B. Widrow,et al.  Stationary and nonstationary learning characteristics of the LMS adaptive filter , 1976, Proceedings of the IEEE.

[2]  Ali H. Sayed,et al.  Recursive linear estimation in Krein spaces. II. Applications , 1993, Proceedings of 32nd IEEE Conference on Decision and Control.

[3]  Keith Glover,et al.  Derivation of the maximum entropy H ∞-controller and a state-space formula for its entropy , 1989 .

[4]  K. Glover,et al.  State-space formulae for all stabilizing controllers that satisfy and H ∞ norm bound and relations to risk sensitivity , 1988 .

[5]  Pramod P. Khargonekar,et al.  FILTERING AND SMOOTHING IN AN H" SETTING , 1991 .

[6]  P. Whittle.  Risk-Sensitive Optimal Control , 1990 .

[7]  Thomas Kailath,et al.  Linear Systems , 1980 .

[8]  A. Jazwinski.  Stochastic Processes and Filtering Theory , 1970 .

[9]  D. Jacobson,et al.  Optimization of stochastic linear systems with additive measurement and process noise using exponential performance criteria , 1974 .

[10]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[11]  P. Khargonekar,et al.  State-space solutions to standard H/sub 2/ and H/sub infinity / control problems , 1989 .

[12]  U. Shaked,et al.  H,-OPTIMAL ESTIMATION: A TUTORIAL , 1992 .

[13]  Ali H. Sayed,et al.  A state-space approach to adaptive filtering , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[14]  Rhodes,et al.  Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games , 1973 .

[15]  H. Kwakernaak.  A polynomial approach to minimax frequency domain optimization of multivariable feedback systems , 1986 .

[16]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[17]  Michael J. Grimble,et al.  Polynomial Matrix Solution of the H/Infinity/ Filtering Problem and the Relationship to Riccati Equation State-Space Results , 1993, IEEE Trans. Signal Process..

[18]  Tamer Bąar.  Optimum performance levels for minimax filters, predictors and smoothers , 1991 .

[19]  Ali H. Sayed,et al.  Recursive linear estimation in Krein spaces. I. Theory , 1993, Proceedings of 32nd IEEE Conference on Decision and Control.

[20]  Bernard Widrow,et al.  Adaptive switching circuits , 1988 .

[21]  J. Speyer,et al.  Optimal stochastic estimation with exponential cost criteria , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[22]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[23]  T. Kailath,et al.  A state-space approach to adaptive RLS filtering , 1994, IEEE Signal Processing Magazine.