论文信息 - Acceleration of stochastic approximation by averaging

Acceleration of stochastic approximation by averaging

A new recursive algorithm of stochastic approximation type with the averaging of trajectories is investigated. Convergence with probability one is proved for a variety of classical optimization and identification problems. It is also demonstrated for these problems that the proposed algorithm achieves the highest possible rate of convergence.

Boris Polyak | A. Juditsky

[1] J. Kiefer,et al. Stochastic Estimation of the Maximum of a Regression Function , 1952 .

[2] D. Sakrison. Stochastic Approximation: A Recursive Method for Solving Regression Problems1 , 1966 .

[3] J. H. Venter. An extension of the Robbins-Monro procedure , 1967 .

[4] Tamio Shimizu,et al. A Stochastic Approximation Method for Optimization Problems , 1969, Journal of the ACM.

[5] V. Fabian. Asymptotically Efficient Stochastic Approximation; The RM Case , 1973 .

[6] Kumpati S. Narendra,et al. Adaptation and learning in automatic systems , 1974 .

[7] Mikhail Borisovich Nevelʹson,et al. Stochastic Approximation and Recursive Estimation , 1976 .

[8] R. Has’minskiĭ,et al. Stochastic Approximation and Recursive Estimation , 1976 .

[9] Yuri Ermoliev,et al. Stochastic Programming Methods , 1976 .

[10] A. Ruszczynski,et al. Stochastic approximation method with gradient averaging for unconstrained problems , 1983 .

[11] Lennart Ljung,et al. Theory and Practice of Recursive Identification , 1983 .

[12] Graham C. Goodwin,et al. Adaptive filtering prediction and control , 1984 .

[13] D. Ruppert. A Newton-Raphson Version of the Multivariate Robbins-Monro Procedure , 1985 .

[14] A. Feuer,et al. Convergence analysis of smoothed stochastic gradient-type algorithm , 1987 .

[15] Han-Fu Chen,et al. Asymptotically efficient stochastic approximation , 1993 .