Stochastic recursive inclusions with non-additive iterate-dependent Markov noise

Abstract In this paper we study the asymptotic behaviour of stochastic approximation schemes with set-valued drift function and non-additive iterate-dependent Markov noise. We show that a linearly interpolated trajectory of such a recursion is an asymptotic pseudotrajectory for the flow of a limiting differential inclusion obtained by averaging the set-valued drift function of the recursion w.r.t. the stationary distributions of the Markov noise. The limit set theorem by Benaim is then used to characterize the limit sets of the recursion in terms of the dynamics of the limiting differential inclusion. We then state two variants of the Markov noise assumption under which the analysis of the recursion is similar to the one presented in this paper. Scenarios where our recursion naturally appears are presented as applications. These include controlled stochastic approximation, subgradient descent, approximate drift problem and analysis of discontinuous dynamics all in the presence of non-additive iterate-dependent Markov noise.

[1]  M. Benaïm.  A Dynamical System Approach to Stochastic Approximations , 1996 .

[2]  V. Tadić.  Stochastic approximation with random truncations, state-dependent noise and discontinuous dynamics , 1998 .

[3]  Shalabh Bhatnagar,et al.  A Generalization of the Borkar-Meyn Theorem for Stochastic Recursive Inclusions , 2015, Math. Oper. Res..

[4]  G. Fort,et al.  Convergence of Markovian Stochastic Approximation with Discontinuous Dynamics , 2014, SIAM J. Control. Optim..

[5]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[6]  Panos M. Pardalos,et al.  Convex optimization theory , 2010, Optim. Methods Softw..

[7]  M. Metivier,et al.  Applications of a Kushner and Clark lemma to general classes of stochastic algorithms , 1984, IEEE Trans. Inf. Theory.

[8]  Jean-Pierre Aubin,et al.  Viability theory , 1991 .

[9]  J. Aubin,et al.  Differential inclusions set-valued maps and viability theory , 1984 .

[10]  D. Leslie,et al.  Asynchronous stochastic approximation with differential inclusions , 2011, 1112.2288.

[11]  Sean P. Meyn,et al.  The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning , 2000, SIAM J. Control. Optim..

[12]  C. Andrieu,et al.  Markovian stochastic approximation with expanding projections , 2011, 1111.5421.

[13]  M. Benaïm.  Dynamics of stochastic approximation algorithms , 1999 .

[14]  V. Borkar.  Probability Theory: An Advanced Course , 1995 .

[15]  Vivek S. Borkar,et al.  Stochastic approximation with 'controlled Markov' noise , 2006, Systems & control letters (Print).

[16]  V. Kreinovich,et al.  Limit Theorems and Applications of Set-Valued and Fuzzy Set-Valued Random Variables , 2002 .

[17]  A. Nagurney,et al.  Projected Dynamical Systems and Variational Inequalities with Applications , 1995 .

[18]  Vivek S. Borkar,et al.  Optimal Control of Diffusion Processes , 1989 .

[19]  V. Borkar.  Stochastic Approximation: A Dynamical Systems Viewpoint , 2008 .

[20]  Josef Hofbauer,et al.  Stochastic Approximations and Differential Inclusions , 2005, SIAM J. Control. Optim..

[21]  John N. Tsitsiklis,et al.  Analysis of temporal-difference learning with function approximation , 1996, NIPS 1996.