Stochastic approximation: a survey

Stochastic recursive algorithms, also known as stochastic approximation, take many forms and have numerous applications. It is the asymptotic properties that are of interest. The early history, starting with the work of Robbins and Monro, is discussed. An approach to proofs of convergence with probability one is illustrated by a stability‐type argument. For general noise processes and algorithms, the most powerful current approach is what is called the ordinary differential equations (ODE) method. The algorithm is interpolated into a continuous‐time process, which is shown to converge to the solution of an ODE, whose asymptotic properties are those of the algorithm. There are probability one and weak convergence methods, the latter being the easiest to use and the most powerful. After discussing the basic ideas and giving some standard proofs, extensions are outlined. These include multiple time scales, tracking of time changing systems, state‐dependent noise, rate of convergence, and random direction methods for high‐dimensional problems. Copyright © 2009 John Wiley & Sons, Inc.

[1]  R. Bass,et al.  Review: P. Billingsley, Convergence of probability measures , 1971 .

[2]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[3]  Harold J. Kushner,et al.  Adaptive optimization of least-squares tracking algorithms: with applications to adaptive antenna arrays for randomly time-varying mobile communications systems , 2005, IEEE Transactions on Automatic Control.

[4]  S. Ethier,et al.  Markov Processes: Characterization and Convergence , 2005 .

[5]  H. Kushner,et al.  Stochastic Approximation and Recursive Algorithms and Applications , 2003 .

[6]  T. Lai.  Stochastic approximation: invited paper , 2003 .

[7]  Harold J. Kushner,et al.  Rate of Convergence for Constrained Stochastic Approximation Algorithms , 2001, SIAM J. Control. Optim..

[8]  Han-Fu Chen.  Stochastic approximation and its applications , 2002 .

[9]  Andrew J. Heunis,et al.  Law of the iterated logarithm for a constant-gain linear stochastic gradient algorithm , 2000, Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No.01CH37228).

[10]  Tamer Basar,et al.  Analysis of Recursive Stochastic Algorithms , 2001 .

[11]  H. Kushner.  Numerical Methods for Stochastic Control Problems in Continuous Time , 2000 .

[12]  Sean P. Meyn,et al.  The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning , 2000, SIAM J. Control. Optim..

[13]  László Gerencsér,et al.  Convergence rate of moments in stochastic approximation with simultaneous perturbation gradient approximation and resetting , 1999, IEEE Trans. Autom. Control..

[14]  M. Benaïm.  Dynamics of stochastic approximation algorithms , 1999 .

[15]  J. Spall.  Adaptive stochastic approximation by the simultaneous perturbation method , 1998, Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171).

[16]  S. Bhatnagar,et al.  A two timescale stochastic approximation scheme for simulation-based parametric optimization , 1998 .

[17]  J. Spall,et al.  Optimal random perturbations for stochastic approximation using a simultaneous perturbation gradient approximation , 1997, Proceedings of the 1997 American Control Conference (Cat. No.97CH36041).

[18]  Andrew J. Heunis,et al.  Strong diffusion approximations for recursive stochastic algorithms , 1997, IEEE Trans. Inf. Theory.

[19]  E. Chong,et al.  EQUIVALENT NECESSARY AND SUFFICIENT CONDITIONS ON NOISE SEQUENCES FOR STOCHASTIC APPROXIMATION ALGORITHMS , 1996 .

[20]  M. Benaïm.  A Dynamical System Approach to Stochastic Approximations , 1996 .

[21]  H. Kushner,et al.  Stochastic Approximation Methods for Systems Over an InfiniteHorizon , 1996 .

[22]  H. Kushner,et al.  Analysis of adaptive step-size SA algorithms for parameter tracking , 1995, IEEE Trans. Autom. Control..

[23]  Bernard Delyon,et al.  Asymptotical Study of Parameter Tracking Algorithms , 1995 .

[24]  H. Kushner,et al.  Stochastic approximation with averaging and feedback: rapidly convergent "on-line" algorithms , 1995, IEEE Trans. Autom. Control..

[25]  H. Kushner,et al.  Stochastic Approximation with Averaging and Feedback: Faster Convergence , 1995 .

[26]  Lennart Ljung,et al.  Performance analysis of general tracking algorithms , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[27]  Lei Guo.  Stability of recursive stochastic tracking algorithms , 1993, Proceedings of 32nd IEEE Conference on Decision and Control.

[28]  Han-Fu Chen,et al.  Asymptotically efficient stochastic approximation , 1993 .

[29]  H. Kushner,et al.  Stochastic approximation with averaging of the iterates: Optimal asymptotic rate of convergence for , 1993 .

[30]  A. Juditsky.  A stochastic estimation algorithm with observation averaging , 1993, IEEE Trans. Autom. Control..

[31]  Lennart Ljung,et al.  Tracking performance analysis of the forgetting factor RLS algorithm , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[32]  Boris Polyak,et al.  Acceleration of stochastic approximation by averaging , 1992 .

[33]  Bernard Delyon,et al.  Stochastic optimization with averaging of trajectories , 1992 .

[34]  J. Spall.  Multivariate stochastic approximation using a simultaneous perturbation gradient approximation , 1992 .

[35]  J. Brossier.  Egalisation adaptative et estimation de phase : application aux communications sous-marines , 1992 .

[36]  G. Yin.  Stochastic Approximation Via Averaging: The Polyak’s Approach Revisited , 1992 .

[37]  G. Yin.  On extensions of Polyak's averaging approach to stochastic approximation , 1991 .

[38]  Victor Solo,et al.  The limiting behavior of LMS , 1989, IEEE Trans. Acoust. Speech Signal Process..

[39]  P. Dupuis,et al.  Stochastic approximation and large deviations: upper bounds and w.p.1 convergence , 1989 .

[40]  Lennart Ljung,et al.  Frequency domain tracking characteristics of adaptive algorithms , 1989, IEEE Trans. Acoust. Speech Signal Process..

[41]  P. Dupuis,et al.  Asymptotic behavior of constrained stochastic approximations via the theory of large deviations , 1987 .

[42]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[43]  W. Grassman.  Approximation and Weak Convergence Methods for Random Processes with Applications to Stochastic Systems Theory (Harold J. Kushner) , 1986 .

[44]  H. Chen,et al.  STOCHASTIC APPROXIMATION PROCEDURES WITH RANDOMLY VARYING TRUNCATIONS , 1986 .

[45]  S. Thomas Alexander,et al.  Adaptive Signal Processing , 1986, Texts and Monographs in Computer Science.

[46]  P. Kumar,et al.  Theory and practice of recursive identification , 1985, IEEE Transactions on Automatic Control.

[47]  P. Dupuis,et al.  Stochastic Approximations via Large Deviations: Asymptotic Properties , 1985 .

[48]  H. Kushner,et al.  An Invariant Measure Approach to the Convergence of Stochastic Approximations with State Dependent Noise. , 1984 .

[49]  P. Holmes,et al.  Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields , 1983, Applied Mathematical Sciences.

[50]  T. M. Williams,et al.  Optimizing Methods in Statistics , 1981 .

[51]  A. Benveniste,et al.  Robust identification of a nonminimum phase system: Blind adjustment of a linear equalizer in data communications , 1980 .

[52]  Harold J. Kushner.  Diffusion approximations to output processes of nonlinear systems with wide-band inputs and applications , 1980, IEEE Trans. Inf. Theory.

[53]  H. Kushner,et al.  RATES OF CONVERGENCE FOR STOCHASTIC APPROXIMATION TYPE ALGORITHMS , 1979 .

[54]  H. Kushner.  Rates of Convergence for Sequential Monte Carlo Optimization Methods , 1978 .

[55]  R. Has’minskiĭ,et al.  Stochastic Approximation and Recursive Estimation , 1976 .

[56]  Mikhail Borisovich Nevelʹson,et al.  Stochastic Approximation and Recursive Estimation , 1976 .

[57]  Kumpati S. Narendra,et al.  Adaptation and learning in automatic systems , 1974 .

[58]  H. Robbins,et al.  A CONVERGENCE THEOREM FOR NON NEGATIVE ALMOST SUPERMARTINGALES AND SOME APPLICATIONS**Research supported by NIH Grant 5-R01-GM-16895-03 and ONR Grant N00014-67-A-0108-0018. , 1971 .

[59]  F. Downton.  Stochastic Approximation , 1969, Nature.

[60]  M. T. Wasan.  Stochastic Approximation , 1969 .

[61]  D. Sakrison.  A Continuous Kiefer-Wolfowitz Procedure for Random Processes , 1964 .

[62]  J. Sacks.  Asymptotic Distribution of Stochastic Approximation Procedures , 1958 .

[63]  J. Blum.  Multidimensional Stochastic Approximation Methods , 1954 .

[64]  J. Kiefer,et al.  Stochastic Estimation of the Maximum of a Regression Function , 1952 .