Path integral reinforcement learning

Reinforcement learning is one of the most fundamental frameworks of learning control, but applying it to high dimensional control systems, e.g., humanoid robots, has largely been impossible so far. Among the key problems are that classical value function-based approaches run into severe limitations in continuous state-action spaces due to issues of function approximation of value functions, and, moreover, that the computational complexity and time of exploring high dimensional state-action spaces quickly exceeds practical feasibility. As an alternative, researchers have turned to trajectorybased reinforcement learning, which sacrifices global optimality in favor of being applicable to high-dimensional state-action spaces. Model-based approches, inspired by ideas of differential dynamic programming, have demonstrated some sucess if models are accurate, but model-free trajectory-based reinforcement learning has been limited by problems of slow learning and the need to tune many open parameters. In this paper, we review some recent developments of trajectory-based reinforcement learning using the framework of stochastic optimal control with path integrals. The path integral control approach transforms the optimal control problem into an estimation problem based on Monte-Carolo evaluations of a path integral. Based on this idea, a new reinforcement learning algorithm can be derived, called Policy Improvement with Path Integrals (PI2). PI2 is surprising simple and works as a black box learning system, i.e., without the need for manual parameter tuning. Moreover, it learns fast and efficiently in very high dimensional problems, as we demonstrate in a variety of robotic tasks. Interestingly, PI2 can be applied in model-free, hybrid, and model-based scenarios. Given its solid foundation in stochastic optimal control, path integral reinforcement learning offers a wide range of applications of reinforcement learning to very complex and new domains.