Markov Decision Processes: Concepts and Algorithms

Situated in between supervised learning and unsupervised learning, the paradigm of reinforcement learning deals with learning in sequential decision making problems in which there is limited feedback. This text introduces the intuitions and concepts behind Markov decision processes and two classes of algorithms for computing optimal behaviors: reinforcement learning and dynamic programming. First the formal framework of Markov decision process is defined, accompanied by the definition of value functions and policies. The main part of this text deals with introducing foundational classes of algorithms for learning optimal behaviors, based on various definitions of optimality with respect to the goal of learning sequential decisions. Additionally, it surveys efficient extensions of the foundational algorithms, differing mainly in the way feedback given by the environment is used to speed up learning, and in the way they concentrate on relevant parts of the problem. For both model-based and model-free settings these efficient extensions have shown useful in scaling up to larger problems. M ARKOV DECISION PROCESSES (MDP) (Puterman, 1994) are an intuitive and fundamental formalism for decision-theoretic planning (DTP) (Boutilier et al., 1999; Boutilier, 1999), reinforcement learning (RL) (Bertsekas and Tsitsiklis, 1996; Sutton and Barto, 1998; Kaelbling et al., 1996) and other learning problems in stochastic domains. In this model, an environment is modelled as a set of states and actions can be performed to control the system’s state. The goal is to control the system in such a way that some performance criterium is maximized. Many problems such as (stochastic) planning problems, learning robot control and game playing problems have successfully been modelled in terms of an MDP. In fact MDPs have become the de facto standard formalism for learning sequential decision making. DTP (Boutilier et al., 1999), e.g. planning using decision-theoretic notions to represent uncertainty and plan quality, is an important extension of the AI planning paradigm, adding the ability to deal with uncertainty in action effects and the ability to deal with less-defined goals. Furthermore it adds a significant dimension in that it considers situations in which factors such as resource consumption and uncertainty demand solutions of varying quality, for example in real-time decision situations. There are many connections between AI planning, research done in the field of operations research (Winston, 1991) and control theory (Bertsekas, 1995), as most work in these fields on sequential decision making can be viewed as instances of MDPs. The notion of a plan in AI planning, i.e. a series of actions from a start state to a goal state, is extended to the notion of a policy, which is mapping from all states to an (optimal) action, based on decision-theoretic measures of optimality with respect to some goal to be optimized. As an example, consider a typical planning domain, involving boxes to be moved around and where the goal is to move some particular boxes to a designated area. This type of problems can be solved using AI planning techniques. Consider now a slightly more realistic extension in which some of the actions can fail, or have uncertain side-effects that can depend on factors beyond the ∗Compiled from draft material from ”The Logic of Adaptive Behavior” by Martijn van Otterlo (van Otterlo, 2008).