Gradient-Based Optimization of Markov Reward Processes: Practical Variants

We consider a discrete time, nite state Markov reward process that depends on a set of parameters. In earlier work, we proposed a class of (stochastic) gradient descent methods that tune the parameters in order to optimize the average reward, using a single (possibly simulated) sample path of the process of interest. The resulting algorithms can be implemented online, and have the property that the gradient of the average reward converges to zero with probability 1. There is a drawback, however, in that the updates can have a high variance, resulting in slow convergence. In this paper, we address this issue and propose two approaches to reduce the variance which, however, introduce an additional bias into the update direction. We derive bounds for the resulting bias term and characterize the asymptotic behavior of the gradient of the average reward. For one of the approaches considered, the magnitude of the bias term exhibits an interesting dependence on the mixing time of the underlying Markov chain. We use a call admission control problem to illustrate the performance of one of the algorithms.