Gradient in Continuous Time

Policy search is a method for approximately solving an optim al control problem by performing a parametric optimization search in a given class of paramet eriz d policies. In order to process a local optimization technique, such as a gradient method, w e wish to evaluate the sensitivity of the performance measure with respect to the policy paramete rs, th so-calledpolicy gradient . This paper is concerned with the estimation of the policy gradien t for continuous-time, deterministic state dynamics, in a reinforcement learningframework, that is, when the decision maker does not have a model of the state dynamics. We show that usual likelihood ratio methods used in discrete -tim , fail to proceed the gradient because they are subject to variance explosion when the disc retization time-step decreases to 0. We describe an alternative approach based on the approximat ion of the pathwise derivative, which leads to a policy gradient estimate that converges almost su rely to the true gradient when the timestep tends to 0. The underlying idea starts with the derivati on of an explicit representation of the policy gradient using pathwise derivation. This derivatio n makes use of the knowledge of the state dynamics. Then, in order to estimate the gradient from the ob s rvable data only, we use a stochastic policy to discretize the continuous deterministic system i nto a stochastic discrete process, which enables to replace the unknown coefficients by quantities th at solely depend on known data. We prove the almost sure convergence of this estimate to the tru e policy gradient when the discretization time-step goes to zero. The method is illustrated on two target problems, in discret e and continuous control spaces.