Direct Gradient-Based Reinforcement Learning: I. Gradient Estimation Algorithms

Despite their many empirical successes, approximate value -function based approaches to reinforcement learning suffer from a paucity of theoretical guarantees on the performance of the policy generated by the value-func tio . In this paper we pursue an alternative approach: first compute the gradien t of the average reward with respect to the parameters controlling the state transi tio in a Markov chain (be they parameters of a class of approximate value fun ctions generating a policy by some form of look-ahead, or parameters directly pa rameterizing a set of policies), and then use gradient ascent to generate a new set of parameters with increased average reward. We call this method “direct” rein forcement learning because we are not attempting to first find an accurate value-fun ction from which to generate a policy, we are instead adjusting the parameters t o directly improve the average reward. We present an algorithm for computing approximations to the gradient of the average reward from a single sample path of the underlying Ma rkov chain. We show that the accuracy of these approximations depends on th e relationship between the discount factor used by the algorithm and the mixin g time of the Markov chain, and that the error can be made arbitrarily small by set ting he discount factor suitably close to1. We extend this algorithm to the case of partially observabl e Markov decision processes controlled by stochastic polici es. We prove that both algorithms converge with probability 1.