Self Learning Control of Constrained Markov Decision Processes - A Gradient Approach

We present stochastic approximation algorithms for computing the locally optimal policy of a constrained average cost finite state Markov Decision process. Because the optimal control strategy is known to be a randomized policy, we consider here a parameterization of the action probabilities to establish the optimization problem. The stochastic approximation algorithms require computation of the gradient of the cost function with respect to the parameter that characterizes the randomized policy. This is computed by novel simulation based gradient estimation schemes involving weak derivatives. Similar to neuro-dynamic programming algorithms (e.g. Q-learning or Temporal Difference methods), the algorithms proposed in this paper are simulation based and do not require explicit knowledge of the underlying parameters such as transition probabilities. However, unlike neuro-dynamic programming methods, the algorithms proposed here can handle constraints and time varying parameters. Numerical examples are given to illustrate the performance of the algorithms. Resume Nous considerons le probleme du controle optimale des châines de Markov commandees (MDP) avec des contraintes. Sous la randomisation des actions, le probleme est parametrise et peut se re-ecrire en terme d’un probleme d’optimisation non-lineaire avec des contraintes non -lineaires, pour lequel on applique l’approximation stochastique. Celle-ci a besoin de calculer certains gradients par rapport aux parametres de controle. Nous proposons une nouvelle methode pour l’estimationde tels gradients avec des derivees faibles. Notre methode est robuste (comme les algorithmes de Q-learning et des differences temporales) et ne suppose pas que les probabilites de transition soient connues. Par ailleurs, notre methode peut etre appliquee lors qu’il y a des contraintes, contrairement aux autres methodes mentionees. Nous presentons aussi des exemples numeriques pour illustrer la performance de nos algorithmes. Acknowledgments: This work was done while the first author was on leave at the Department of Electrical and Electronic Engineering at the University of Melbourne. The research was supported by the Australian Research Council and research grants from NSERC, Canada and FCAR, Quebec. Les Cahiers du GERAD G–2003–51 1