Similarities and differences between policy gradient methods and evolution strategies

Natural policy gradient methods and the covariance matrix adaptation evolution strategy, two variable metric methods proposed for solving reinforcement learning tasks, are contrasted to point out their con- ceptual similarities and differences. Experiments on the cart pole bench- mark are conducted as a first attempt to compare their performance. Reinforcement learning (RL) algorithms search for a policy mapping states of the environment to (a probability distribution over) the actions an agent can take in those states. The goal is to find a behavior such that some notion of future reward is maximized. Direct policy search methods address this task by directly learning parameters of a function explicitly representing the policy. Here we consider two general approaches to conduct direct policy search, namely policy gradient methods (PGMs) and evolution strategies (ESs). We will argue that these approaches are quite similar. This makes it all the more surprising that so far there has been no systematic comparison of PGMs and ESs applied to the same test problems and operating on the same class of policies with the same parameterization. This paper is our attempt to draw such a comparison, on a conceptual level and by conducting first empirical studies. We restrict our consideration to the natural actor critic algorithm (NAC, (1, 2)) and the covari- ance matrix adaptation ES (CMA-ES, (3)), which are compared in the context of optimization in (4). We picked these two because they can be considered state- of-the-art, they are our favorite direct policy search method and evolutionary RL algorithm, respectively, and they are both variable metric methods. In section 2 we briefly review the NAC algorithm and the CMA-ES. Section 3 describes the conceptual relations of these two approaches and in section 4 we use a simple toy problem to compare the methods empirically.