Using Generative Adversarial Nets on Atari Games for Feature Extraction in Deep Reinforcement Learning

Deep Reinforcement Learning (DRL) has been successfully applied in several research domains such as robot navigation and automated video game playing. However, these methods require excessive computation and interaction with the environment, so enhancements on sample efficiency are required. The main reason for this requirement is that sparse and delayed rewards do not provide an effective supervision for representation learning of deep neural networks. In this study, Proximal Policy Optimization (PPO) algorithm is augmented with Generative Adversarial Networks (GANs) to increase the sample efficiency by enforcing the network to learn efficient representations without depending on sparse and delayed rewards as supervision. The results show that an increased performance can be obtained by jointly training a DRL agent with a GAN discriminator. ---- Derin Pekistirmeli Ogrenme, robot navigasyonu ve otomatiklestirilmis video oyunu oynama gibi arastirma alanlarinda basariyla uygulanmaktadir. Ancak, kullanilan yontemler ortam ile fazla miktarda etkilesim ve hesaplama gerektirmekte ve bu nedenle de ornek verimliligi yonunden iyilestirmelere ihtiyac duyulmaktadir. Bu gereksinimin en onemli nedeni, gecikmeli ve seyrek odul sinyallerinin derin yapay sinir aglarinin etkili betimlemeler ogrenebilmesi icin yeterli bir denetim saglayamamasidir. Bu calismada, Proksimal Politika Optimizasyonu algoritmasi Uretici Cekismeli Aglar (UCA) ile desteklenerek derin yapay sinir aglarinin seyrek ve gecikmeli odul sinyallerine bagimli olmaksizin etkili betimlemeler ogrenmesi tesvik edilmektedir. Elde edilen sonuclar onerilen algoritmanin ornek verimliliginde artis elde ettigini gostermektedir.

[1]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[2]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[3]  Francesco Visin,et al.  A guide to convolution arithmetic for deep learning , 2016, ArXiv.

[4]  Geoffrey E. Hinton.  Reducing the Dimensionality of Data with Neural , 2008 .

[5]  Daiki Kimura,et al.  DAQN: Deep Auto-encoder and Q-Network , 2018, ArXiv.

[6]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[7]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[8]  Tom Schaul,et al.  Rainbow: Combining Improvements in Deep Reinforcement Learning , 2017, AAAI.

[9]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[10]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[11]  Marc G. Bellemare,et al.  The Arcade Learning Environment: An Evaluation Platform for General Agents , 2012, J. Artif. Intell. Res..

[12]  Martin A. Riedmiller,et al.  Deep auto-encoder neural networks in reinforcement learning , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[13]  Alexia Jolicoeur-Martineau,et al.  The relativistic discriminator: a key element missing from standard GAN , 2018, ICLR.

[14]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[15]  Yoshua Bengio,et al.  Why Does Unsupervised Pre-training Help Deep Learning? , 2010, AISTATS.