Sample-Efficient Deep RL with Generative Adversarial Tree Search

We propose Generative Adversarial Tree Search (GATS), a sample-efficient Deep Reinforcement Learning (DRL) algorithm. While Monte Carlo Tree Search (MCTS) is known to be effective for search and planning in RL but, it is often sampleinefficient and therefore expensive to apply in practice. In this work, we train Generative Adversarial Networks (GANs) to model an environment’s dynamics and train a predictor to learn the reward function. While typical DRL algorithms estimate the Q function or optimize directly over a parameterized policy, we exploit the collected data through interaction with the environment to train both a reward predictor conditional GAN that simulated state transitions. During planning, we deploy finite depth MCTS, using the trained generative model for the tree search and estimated Q value for the leaves, in order to find the best policy. We theoretically show that GATS improves the bias-variance trade-off in DRL. On the Atari game Pong, GATS significantly reduces the bias in Q estimates and leads to a drastic reduction of sample complexity of DQN by a factor of 200%.