Bootstrapping learning from abstract models in games

Computer gaming environments are real time, dynamic, and complex, with incomplete knowledge of the world. Agents in such environments require detailed models of the world if they are to learn effective policies. Machine learning techniques such as reinforcement learning can become intractably large, detailed world models. In this paper we tackle the well-known problem of low convergence speed in reinforcement learning for the detailed model of the world, specifically for video games. We propose first training the agents with an abstract model of the world and then using the resulting policy to initialise the system prior to training the agent with the detailed model of the world. This paper reports on results from applying the proposed technique to the classic arcade game Asteroids. Our experiments show that an agent can quickly learn a policy with the abstract model, and that when this policy's learned values are used to initialise the detailed model, learning with the detailed model improves the rate of convergence.

[1]  Marcus Gallagher,et al.  Learning to be a Bot: Reinforcement Learning in Shooter Games , 2008, AIIDE.

[2]  Juan Julián Merelo Guervós,et al.  Evolving the cooperative behaviour in Unreal™ bots , 2010, Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games.

[3]  Philip Hingston,et al.  Bots trained to play like a human are more fun , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[4]  Daniele Loiacono,et al.  Learning to overtake in TORCS using simple reinforcement learning , 2010, IEEE Congress on Evolutionary Computation.

[5]  Norman Carver,et al.  Tuning computer gaming agents using Q-learning , 2011, 2011 Federated Conference on Computer Science and Information Systems (FedCSIS).

[6]  Andrew N. Marshall,et al.  Gamebots: A 3D Virtual World Test-Bed For Multi-Agent Research , 2001 .

[7]  Anna I. Esparcia-Alcázar,et al.  Controlling bots in a First Person Shooter game using genetic algorithms , 2010, IEEE Congress on Evolutionary Computation.

[8]  Sushil J. Louis,et al.  Using a genetic algorithm to tune first-person shooter bots , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[9]  Hector Muñoz-Avila,et al.  RETALIATE: Learning Winning Policies in First-Person Shooter Games , 2007, AAAI.

[10]  Simon Colton,et al.  Combining AI Methods for Learning Bots in a Real-Time Strategy Game , 2009, Int. J. Comput. Games Technol..

[11]  David M. Bourg,et al.  AI for Game Developers , 2004 .

[12]  Ah-Hwee Tan,et al.  Integrating Temporal Difference Methods and Self-Organizing Neural Networks for Reinforcement Learning With Delayed Evaluative Feedback , 2008, IEEE Transactions on Neural Networks.

[13]  Anastasios Tefas,et al.  Neural Networks Training for Weapon Selection in First-Person Shooter Games , 2010, ICANN.

[14]  Philip Hingston,et al.  A new design for a Turing Test for Bots , 2010, Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games.

[15]  Thore Graepel,et al.  LEARNING TO FIGHT , 2004 .

[16]  Purvag Patel,et al.  Improving Computer Game Bots' behavior using Q-Learning , 2009 .

[17]  Yang Gao,et al.  RL-DOT: A Reinforcement Learning NPC Team for Playing Domination Games , 2010, IEEE Transactions on Computational Intelligence and AI in Games.

[18]  Cédric Buche,et al.  The Challenge of Believability in Video Games: Definitions, Agents Models and Imitation Learning , 2010, CGAMES 2010.

[19]  Marcus Gallagher,et al.  Reinforcement Learning in First Person Shooter Games , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[20]  Ah-Hwee Tan,et al.  Creating Human-like Autonomous Players in Real-time First Person Shooter Computer Games , 2009 .

[21]  Ah-Hwee Tan,et al.  FALCON: a fusion architecture for learning, cognition, and navigation , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).