Hierarchical controller learning in a First-Person Shooter

We describe the architecture of a hierarchical learning-based controller for bots in the First-Person Shooter (FPS) game Unreal Tournament 2004. The controller is inspired by the subsumption architecture commonly used in behaviourbased robotics. A behaviour selector decides which of three sub-controllers gets to control the bot at each time step. Each controller is implemented as a recurrent neural network, and trained with artificial evolution to perform respectively combat, exploration and path following. The behaviour selector is trained with a multiobjective evolutionary algorithm to achieve an effective balancing of the lower-level behaviours. We argue that FPS games provide good environments for studying the learning of complex behaviours, and that the methods proposed here can help developing interesting opponents for games.

[1]  John Levine,et al.  Scaling-up behaviours in EvoTanks: Applying subsumption principles to artificial neural networks , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[2]  Julian Togelius,et al.  Generating diverse opponents with multiobjective evolution , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[3]  Ronald C. Arkin,et al.  An Behavior-based Robotics , 1998 .

[4]  Christian Bauckhage,et al.  Learning Human-Like Movement Behavior for Computer Games , 2004 .

[5]  Phil Husbands,et al.  Evolutionary robotics , 2014, Evolutionary Intelligence.

[6]  Oliver Kramer,et al.  Evolution of Human-Competitive Agents in Modern Computer Games , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[7]  Sushil J. Louis,et al.  Using a genetic algorithm to tune first-person shooter bots , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[8]  Jörg Denzinger,et al.  Dealing with Parameterized Actions in Behavior Testing of Commercial Computer Games , 2005, CIG.

[9]  Joost Westra.  Evolutionary Neural Networks applied in First Person Shooters , 2007 .

[10]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[11]  Jürgen Schmidhuber,et al.  LSTM recurrent networks learn simple context-free and context-sensitive languages , 2001, IEEE Trans. Neural Networks.

[12]  Rudolf Kadlec,et al.  POGAMUT 2 - A PLATFORM FOR FAST DEVELOPMENT OF VIRTUAL AGENTS' BEHAVIOR , 2007 .

[13]  Julian Togelius,et al.  Robust player imitation using multiobjective evolution , 2009, 2009 IEEE Congress on Evolutionary Computation.

[14]  Andrew N. Marshall,et al.  Gamebots: A 3D Virtual World Test-Bed For Multi-Agent Research , 2001 .

[15]  Marcus Gallagher,et al.  Creating a multi-purpose first person shooter bot with reinforcement learning , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[16]  Rudolf Kadlec,et al.  Evoluce chovan´õ inteligentn´õch agentû u v poÿc´õtaÿcovych hrach Evolution of intelligent agent behaviour in computer games , 2008 .

[17]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[18]  Simon D. Levy,et al.  Adding Smart Opponents to a First-Person Shooter Video Game through Evolutionary Design , 2005, AIIDE.

[19]  Risto Miikkulainen,et al.  Integration and Evaluation of Exploration-Based Learning in Games , 2006, 2006 IEEE Symposium on Computational Intelligence and Games.

[20]  Risto Miikkulainen,et al.  Incremental Evolution of Complex General Behavior , 1997, Adapt. Behav..

[21]  Bobby D. Bryant,et al.  Neuro-visual control in the Quake II game engine , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[22]  Julian Togelius,et al.  Evolution of a subsumption architecture neurocontroller , 2004, J. Intell. Fuzzy Syst..