Deep Residual Attention Reinforcement Learning

Making decisions based more on the crucial objects which are closely connected to the reward in a given visual input is advantageous in reinforcement learning. In this work, we incorporate an attention-based structure into the network structure of Importance Weighted Actor-Learner Architecture (IMPALA) to help the model find out the crucial objects and propose Deep Residual Attention Reinforcement Learning (DRARL). Experiments in Atari games and special environments which have more irrelevant objects than usual demonstrate the superiority of DRARL in the multi-objects environment compared to the original IMPALA. Furthermore, the visualization of trained agents’ attention indicates that the additional attention mechanism helps IMPALA concentrate on the crucial objects and therefore improves the performance of IMPALA.

[1]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[2]  Shane Legg,et al.  IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures , 2018, ICML.

[3]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[4]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[5]  Razvan Pascanu,et al.  Relational Deep Reinforcement Learning , 2018, ArXiv.

[6]  Mikhail Pavlov,et al.  Deep Attention Recurrent Q-Network , 2015, ArXiv.

[7]  Dana H. Ballard,et al.  An Initial Attempt of Combining Visual Selective Attention with Deep Reinforcement Learning , 2018, ArXiv.

[8]  Yuan Chang Leong,et al.  Dynamic Interaction between Reinforcement Learning and Attention in Multidimensional Environments , 2017, Neuron.

[9]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[11]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[12]  Fei Sha,et al.  Actor-Attention-Critic for Multi-Agent Reinforcement Learning , 2018, ICML.

[13]  Alex Graves,et al.  Recurrent Models of Visual Attention , 2014, NIPS.

[14]  Zeynep Akata,et al.  Visual Rationalizations in Deep Reinforcement Learning for Atari Games , 2018, BNCAI.

[15]  Xiaogang Wang,et al.  Residual Attention Network for Image Classification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).