Finite-Time Convergence Rates of Nonlinear Two-Time-Scale Stochastic Approximation under Markovian Noise

We study the so-called two-time-scale stochastic approximation, a simulation-based approach for finding the roots of two coupled nonlinear operators. Our focus is to characterize its finite-time performance in a Markov setting, which often arises in stochastic control and reinforcement learning problems. In particular, we consider the scenario where the data in the method are generated by Markov processes, therefore, they are dependent. Such dependent data result to biased observations of the underlying operators. Under some fairly standard assumptions on the operators and the Markov processes, we provide a formula that characterizes the convergence rate of the mean square errors generated by the method to zero. Our result shows that the method achieves a convergence in expectation at a rate O(1/k), where k is the number of iterations. Our analysis is mainly motivated by the classic singular perturbation theory for studying the asymptotic convergence of two-time-scale systems, that is, we consider a Lyapunov function that carefully characterizes the coupling between the two iterates. In addition, we utilize the geometric mixing time of the underlying Markov process to handle the bias and dependence in the data. Our theoretical result complements for the existing literature, where the rate of nonlinear two-time-scale stochastic approximation under Markovian noise is unknown. 1 Nonlinear two-time-scale SA Stochastic approximation (SA), introduced by [1], is a simulation-based approach for finding the root (or fixed point) of some unknown operator F represented by the form of an expectation, i.e., F (x) = Eπ[F (x, ξ)], where ξ is some random variable with a distribution π. Specifically, this method seeks a point x⋆ such that F (x⋆) = 0 based on the noisy observations F (x; ξ). The iterate x is iteratively updated by moving along the direction of F (x; ξ) scaled by some step size. Through a careful choice of this step size, the “noise” induced by the random samples ξ can be averaged out across iterations, and the algorithm converges to x. SA has found broad applications in many areas including statistics, stochastic optimization, machine learning, and reinforcement learning [2, 3, 4]. In this paper, we consider the two-time-scale SA, a generalized variant of the classic SA, which is used to find the root of a system of two coupled nonlinear equations. Given two unknown operators F : Rd×Rd → R d and G : Rd × Rd → Rd represented by F (x, y) = Eπ[F (x, y; ξ)] and G(x, y) = Eπ[G(x, y; ξ)], we seek to find x⋆ and y⋆ such that { F (x, y) = 0 G(x, y) = 0. (1) *Thinh T. Doan is with the Bradley Department of Electrical and Computer Engineering, Virginia Tech, USA. Email: thinhdoan@vt.edu