Stabilization of stochastic approximation by step size adaptation

A scheme for stabilizing stochastic approximation iterates by adaptively scaling the step sizes is proposed and analyzed. This scheme leads to the same limiting differential equation as the original scheme and therefore has the same limiting behavior, while avoiding the difficulties associated with projection schemes. The proof technique requires only that the limiting o.d.e. descend a certain Lyapunov function outside an arbitrarily large bounded set. (C) 2012 Elsevier B.V. All rights reserved.

[1]  V. Borkar.  Probability Theory: An Advanced Course , 1995 .

[2]  John N. Tsitsiklis,et al.  Asynchronous stochastic approximation and Q-learning , 1994, Mach. Learn..

[3]  Eric Moulines,et al.  Stability of Stochastic Approximation under Verifiable Conditions , 2005, Proceedings of the 44th IEEE Conference on Decision and Control.

[4]  Vivek S. Borkar,et al.  On the Lock-in Probability of Stochastic Approximation , 2002, Combinatorics, Probability and Computing.

[5]  Sean P. Meyn,et al.  The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning , 2000, SIAM J. Control. Optim..

[6]  Sigrún Andradóttir,et al.  A Stochastic Approximation Algorithm with Varying Bounds , 1995, Oper. Res..

[7]  Tamer Basar,et al.  Analysis of Recursive Stochastic Algorithms , 2001 .

[8]  Stephen S. Wilson,et al.  Random iterative models , 1996 .

[9]  Pierre Priouret,et al.  Adaptive Algorithms and Stochastic Approximations , 1990, Applications of Mathematics.

[10]  Vivek S. Borkar,et al.  Stochastic Approximation for Nonexpansive Maps: Application to Q-Learning Algorithms , 1997, SIAM J. Control. Optim..

[11]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[12]  V. Borkar.  Stochastic Approximation: A Dynamical Systems Viewpoint , 2008 .

[13]  S. Andradóttir.  A Scaled Stochastic Approximation Algorithm , 1996 .

[14]  M. Benaïm.  Dynamics of stochastic approximation algorithms , 1999 .

[15]  Vijay R. Konda,et al.  OnActor-Critic Algorithms , 2003, SIAM J. Control. Optim..

[16]  Han-Fu Chen.  Stochastic approximation and its applications , 2002 .

[17]  H. Kushner,et al.  Stochastic Approximation and Recursive Algorithms and Applications , 2003 .