Simulation-Based Methods for Markov Decision Processes

Markov decision processes have been a popular paradigm for sequential decision making under uncertainty. Dynamic programming provides a framework for studying such problems, as well as for devising algorithms to compute an optimal control policy. Dynamic programming methods rely on a suitably de ned value function that has to be computed for every state in the state space. However, many interesting problems involve very large state spaces (\curse of dimensionality"), which prohibits the application of dynamic programming . In addition, dynamic programming assumes the availability of an exact model, in the form of transition probabilities (\curse of modeling"). In many practical situations, such a model is not available and one must resort to simulation or experimentation with an actual system. For all of these reasons, dynamic programming in its pure form may be inapplicable. In this thesis we study an approach for overcoming these di culties where we use (a) compact (parametric) representations of the control policy, thus avoiding the curse of dimensionality, and (b) simulation to estimate quantities of interest, thus avoiding model-based computations (curse of modeling). It is not limited to Markov decision processes, but applies to general Markov reward processes for which the transition probabilities and the one-stage rewards depend on a parameter vector . We propose gradient-type algorithms for updating based on the simulation of a single sample path, so as to improve a given performance measure. As possible performance measures we consider the weighted reward-to-go and the average reward. The corresponding algorithms (a) can be implemented online and update the parameter vector either at visits to a certain state, or at every time step. (b) have the property that the gradient (with respect to ) of the performance measure converges to 0 with probability 1. This is the strongest possible result for gradientrelated stochastic approximation algorithms. We illustrate the methodology by considering the call admission control problem where a telecommunications provider sells bandwith of a single communication link to customers so as to optimize the revenue. We use the proposed algorithms to optimize the parameters a heuristic threshold policy for this problem. 2