Counter-Strike Deathmatch with Large-Scale Behavioural Cloning

This paper describes an AI agent that plays the popular first-person-shooter (FPS) video game ‘Counter-Strike; Global Offensive’ (CSGO) from pixel input. The agent, a deep neural network, matches the performance of the medium difficulty built-in AI on the deathmatch game mode, whilst adopting a humanlike play style. Unlike much prior work in games, no API is available for CSGO, so algorithms must train and run in real-time. This limits the quantity of on-policy data that can be generated, precluding many reinforcement learning algorithms. Our solution uses behavioural cloning — training on a large noisy dataset scraped from human play on online servers (4 million frames, comparable in size to ImageNet), and a smaller dataset of high-quality expert demonstrations. This scale is an order of magnitude larger than prior work on imitation learning in FPS games. Gameplay examples: https://youtu.be/p01vWk7uMvM Figure 1: Screenshot, agent’s vision and map overview for the deathmatch game mode.

[1]  Taylor Stanton Hardenstein.  "Skins" in the Game: Counter-Strike, Esports, and the Shady World of Online Gambling , 2017 .

[2]  Guy Lever,et al.  Human-level performance in 3D multiplayer games with population-based reinforcement learning , 2018, Science.

[3]  Dmitry I. Ignatov,et al.  Predicting Winning Team and Probabilistic Ratings in "Dota 2" and "Counter-Strike: Global Offensive" Video Games , 2017, AIST.

[4]  Kristoffer Sjöö,et al.  Imitation Learning with Concurrent Actions in 3D Games , 2018, 2018 IEEE Conference on Computational Intelligence and Games (CIG).

[5]  Mohammad Norouzi,et al.  An Optimistic Perspective on Offline Reinforcement Learning , 2020, ICML.

[6]  Hang Su,et al.  Playing FPS Games With Environment-Aware Hierarchical Reinforcement Learning , 2019, IJCAI.

[7]  Zhao Chen,et al.  The Game Imitation: Deep Supervised Convolutional Networks for Quick Video Game AI , 2017, ArXiv.

[8]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[9]  Jonathan Lee,et al.  Iterative Noise Injection for Scalable Imitation Learning , 2017, ArXiv.

[10]  Peter Bakker,et al.  Robot see, robot do: An overview of robot imitation , 1996 .

[11]  Jakub W. Pachocki,et al.  Dota 2 with Large Scale Deep Reinforcement Learning , 2019, ArXiv.

[12]  Guillaume Lample,et al.  Playing FPS Games with Deep Reinforcement Learning , 2016, AAAI.

[13]  Geoffrey J. Gordon,et al.  A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning , 2010, AISTATS.

[14]  Stefan Schaal,et al.  Learning from Demonstration , 1996, NIPS.

[15]  S. Levine,et al.  Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems , 2020, ArXiv.

[16]  Sergey Levine,et al.  Causal Confusion in Imitation Learning , 2019, NeurIPS.

[17]  Bernard Gorman.  IMITATIVE LEARNING OF COMBAT BEHAVIOURS IN FIRST-PERSON COMPUTER GAMES , 2007 .

[18]  Justin Fu,et al.  Datasets for Data-Driven Reinforcement Learning , 2020 .

[19]  Junxiao Song,et al.  SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II , 2020, ICML.

[20]  Tom Schaul,et al.  Deep Q-learning From Demonstrations , 2017, AAAI.

[21]  Ville Hautamäki,et al.  Benchmarking End-to-End Behavioural Cloning on Video Games , 2020, 2020 IEEE Conference on Games (CoG).

[22]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[23]  Wojciech Jaskowski,et al.  ViZDoom: A Doom-based AI research platform for visual reinforcement learning , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[24]  Santiago Ontañón,et al.  A Dynamic-Bayesian Network framework for modeling and evaluating learning from observation , 2014, Expert Syst. Appl..

[25]  Vadim Bulitko,et al.  An evaluation of models for predicting opponent positions in first-person shooter video games , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[26]  Ting Chen,et al.  Combo-Action: Training Agent For FPS Game with Auxiliary Tasks , 2019, AAAI.

[27]  Ruslan Salakhutdinov,et al.  MineRL: A Large-Scale Dataset of Minecraft Demonstrations , 2019, IJCAI.

[28]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[29]  Wojciech M. Czarnecki,et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning , 2019, Nature.

[30]  Marek Wydmuch,et al.  ViZDoom Competitions: Playing Doom From Pixels , 2018, IEEE Transactions on Games.

[31]  Nicole C. Krämer,et al.  Underlying factors of social capital acquisition in the context of online-gaming: Comparing World of Warcraft and Counter-Strike , 2014, Comput. Hum. Behav..

[32]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.