Introduction

This is the third special issue of Machine Learning on the subject of reinforcement learning (the first and second special issues were edited by Richard Sutton in 1992 and Leslie Kaelbling in 1996 respectively). The field of reinforcement learning continues to grow, attracting ideas and participants not only from AI and machine learning but also from neuroscience, cognitive science, operations research, and control. More than a decade of the resulting research has led to great progress in the theoretical underpinnings of the field, much of it derived from the theory of dynamic programming and the associated frameworks of Markov Decision Processes (MDPs), semi-MDPs and partially observable MDPs (POMDPs) (see texts by Sutton and Barto (1998) and Bertsekas and Tsitsiklis (1996) for excellent overviews). Much has been accomplished, and yet, of course, much remains to be done. I will take advantage of this guest editorial to outline my general views on three of the open issues that are key to further rapid progress in reinforcement learning, and then turn to very briefly survey the papers in this special issue.