Reinforcement Learning in First Person Shooter Games

Reinforcement learning (RL) is a popular machine learning technique that has many successes in learning how to play classic style games. Applying RL to first person shooter (FPS) games is an interesting area of research as it has the potential to create diverse behaviors without the need to implicitly code them. This paper investigates the tabular Sarsa (λ) RL algorithm applied to a purpose built FPS game. The first part of the research investigates using RL to learn bot controllers for the tasks of navigation, item collection, and combat individually. Results showed that the RL algorithm was able to learn a satisfactory strategy for navigation control, but not to the quality of the industry standard pathfinding algorithm. The combat controller performed well against a rule-based bot, indicating promising preliminary results for using RL in FPS games. The second part of the research used pretrained RL controllers and then combined them by a number of different methods to create a more generalized bot artificial intelligence (AI). The experimental results indicated that RL can be used in a generalized way to control a combination of tasks in FPS bots such as navigation, item collection, and combat.

[1]  Gerald Tesauro,et al.  Temporal difference learning and TD-Gammon , 1995, CACM.

[2]  Marcus Gallagher,et al.  Learning to be a Bot: Reinforcement Learning in Shooter Games , 2008, AIIDE.

[3]  Gillian Hayes,et al.  Group utility functions: learning equilibria between groups of agents in computer games by modifying the reinforcement signal , 2005, 2005 IEEE Congress on Evolutionary Computation.

[4]  Eric O. Postma,et al.  TEAM: The Team-Oriented Evolutionary Adaptability Mechanism , 2004, ICEC.

[5]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[6]  Shahar Cohen,et al.  Reinforcement Learning with Hierarchical Decision-Making , 2006, Sixth International Conference on Intelligent Systems Design and Applications.

[7]  Kathryn E. Merrick,et al.  Motivated reinforcement learning for non-player characters in persistent computer game worlds , 2006, ACE '06.

[8]  Manfred Huber,et al.  A hybrid architecture for hierarchical reinforcement learning , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[9]  Nathan R. Sturtevant,et al.  Feature Construction for Reinforcement Learning in Hearts , 2006, Computers and Games.

[10]  John E. Laird,et al.  It knows what you're going to do: adding anticipation to a Quakebot , 2001, AGENTS '01.

[11]  Hector Muñoz-Avila,et al.  RETALIATE: Learning Winning Policies in First-Person Shooter Games , 2007, AAAI.

[12]  Marcus Gallagher,et al.  Creating a multi-purpose first person shooter bot with reinforcement learning , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[13]  Jason Jones.  Benefits of Genetic Algorithms in Simulations for Game Designers , 2003 .

[14]  Kathryn E. Merrick,et al.  Modeling motivation for adaptive nonplayer characters in dynamic computer game worlds , 2008, CIE.

[15]  S. Levy,et al.  Evolving AI Opponents in a First-Person-Shooter Video Game , 2005, AAAI.

[16]  John E. Laird,et al.  Integrating Reinforcement Learning with Soar. , 2004 .

[17]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[18]  John E. Laird,et al.  Soar-RL: integrating reinforcement learning with Soar , 2005, Cognitive Systems Research.

[19]  Thore Graepel,et al.  LEARNING TO FIGHT , 2004 .

[20]  Daniel Sanchez-Crespo Dalmau,et al.  Core Techniques and Algorithms in Game Programming , 2003 .

[21]  Toshihiko Watanabe,et al.  Hierarchical reinforcement learning using a modular fuzzy model for multi-agent problem , 2007, 2007 IEEE International Conference on Systems, Man and Cybernetics.

[22]  Bruce Blumberg,et al.  Integrated learning for interactive synthetic characters , 2002, SIGGRAPH.

[23]  Ah-Hwee Tan,et al.  Self-organizing cognitive agents and reinforcement learning in multi-agent environment , 2005, IEEE/WIC/ACM International Conference on Intelligent Agent Technology.

[24]  Se-Young Oh,et al.  TD based reinforcement learning using neural networks in control problems with continuous action space , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[25]  W. Marsden.  I and J , 2012 .