Real-time neuroevolution in the NERO video game

In most modern video games, character behavior is scripted; no matter how many times the player exploits a weakness, that weakness is never repaired. Yet, if game characters could learn through interacting with the player, behavior could improve as the game is played, keeping it interesting. This paper introduces the real-time Neuroevolution of Augmenting Topologies (rtNEAT) method for evolving increasingly complex artificial neural networks in real time, as a game is being played. The rtNEAT method allows agents to change and improve during the game. In fact, rtNEAT makes possible an entirely new genre of video games in which the player trains a team of agents through a series of customized exercises. To demonstrate this concept, the Neuroevolving Robotic Operatives (NERO) game was built based on rtNEAT. In NERO, the player trains a team of virtual robots for combat against other players' teams. This paper describes results from this novel application of machine learning, and demonstrates that rtNEAT makes possible video games like NERO where agents evolve and adapt in real time. In the future, rtNEAT may allow new kinds of educational and training applications through interactive and adapting games.

[1]  Arthur L. Samuel,et al.  Some Studies in Machine Learning Using the Game of Checkers , 1967, IBM J. Res. Dev..

[2]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[3]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[4]  Terrence J. Sejnowski,et al.  A 'Neural' Network that Learns to Play Backgammon , 1987, NIPS.

[5]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[6]  Alden H. Wright,et al.  Genetic Algorithms for Real Parameter Optimization , 1990, FOGA.

[7]  A. P. Wieland,et al.  Evolving neural network controllers for unstable systems , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[8]  D. R. McGregor,et al.  Designing application-specific neural networks using the structured genetic algorithm , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[9]  Byoung-Tak Zhang,et al.  Evolving Optimal Neural Networks Using Genetic Algorithms with Occam's Razor , 1993, Complex Syst..

[10]  Heinrich Braun,et al.  Evolving Neural Feedforward Networks , 1993 .

[11]  Martin Mandischer,et al.  Representation and Evolution of Neural Networks , 1993 .

[12]  Francesco Mondada,et al.  Automatic creation of an autonomous agent: genetic evolution of a neural-network driven robot , 1994 .

[13]  I. Harvey.  The artificial evolution of adaptive behaviour , 1994 .

[14]  G. Kane.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[15]  Hava T. Siegelmann,et al.  Analog computation via neural networks , 1993, [1993] The 2nd Israel Symposium on Theory and Computing Systems.

[16]  Rajendra Krishnan,et al.  2DELTA-GANN: A NEW APPROACH TO TRAINING NEURAL NETWORKS USING GENETIC ALGORITHMS , 1994 .

[17]  Vittorio Maniezzo,et al.  Genetic evolution of the topology and weight distribution of neural networks , 1994, IEEE Trans. Neural Networks.

[18]  Jeffrey L. Elman,et al.  Learning and Evolution in Neural Networks , 1994, Adapt. Behav..

[19]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[20]  John J. Grefenstette,et al.  A Coevolutionary Approach to Learning Sequential Decision Rules , 1995, ICGA.

[21]  David B. Fogel,et al.  Evolving Neural Control Systems , 1995, IEEE Expert.

[22]  S. Carroll.  Homeotic genes and the evolution of arthropods and chordates , 1995, Nature.

[23]  Risto Miikkulainen,et al.  Learning Sequential Decision Tasks , 1995 .

[24]  David E. Moriarty,et al.  Discovering Complex Othello Strategies through Evolutionary Neural Networks , 1995, Connect. Sci..

[25]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[26]  Pattie Maes,et al.  Evolving Obstacle Avoidance Behavior in a Robot Arm , 1996 .

[27]  Larry D. Pyeatt,et al.  A comparison between cellular encoding and direct encoding for genetic neural networks , 1996 .

[28]  Risto Miikkulainen,et al.  Evolving Obstacle Avoidance Behavior in a Robot Arm , 1996 .

[29]  Jong-Hwan Kim,et al.  Evolutionary ordered neural network with a linked-list encoding scheme , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[30]  Riccardo,et al.  Evolution of the Topology and the Weightsof Neural Networks using GeneticProgramming with a Dual RepresentationJo , 1997 .

[31]  David E. Moriarty,et al.  Symbiotic Evolution of Neural Networks in Sequential Decision Tasks , 1997 .

[32]  Ashwin Ram,et al.  Experiments with Reinforcement Learning in Problems with Continuous State and Action Spaces , 1997, Adapt. Behav..

[33]  David W. Opitz,et al.  Connectionist Theory Refinement: Genetically Searching the Space of Network Topologies , 1997, J. Artif. Intell. Res..

[34]  Risto Miikkulainen,et al.  2-D Pole Balancing with Recurrent Evolutionary Networks , 1998 .

[35]  Shin Ishii,et al.  Strategy Acquisition for the Game "Othello" Based on Reinforcement Learning , 1999, ICONIP.

[36]  Xin Yao,et al.  Towards designing artificial neural networks by evolution , 1998 .

[37]  Kenneth O. Stanley,et al.  Real-time Interactive Neuro-evolution , 1998 .

[38]  Y L Wang,et al.  Zebrafish hox clusters and vertebrate genome evolution. , 1998, Science.

[39]  Andrew P. Martin.  Increasing Genomic Complexity by Gene Duplication and the Origin of Vertebrates , 1999, The American Naturalist.

[40]  X. Yao.  Evolving Artificial Neural Networks , 1999 .

[41]  A. Force,et al.  Preservation of duplicate genes by complementary, degenerative mutations. , 1999, Genetics.

[42]  Risto Miikkulainen,et al.  Solving Non-Markovian Control Tasks with Neuro-Evolution , 1999, IJCAI.

[43]  Ian C. Parmee,et al.  Towards the support of innovative conceptual design through interactive designer/evolutionary computing strategies , 2000, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[44]  Johannes Fürnkranz,et al.  Machine learning in games: a survey , 2001 .

[45]  Eytan Ruppin,et al.  Emergence of Memory-Driven Command Neurons in Evolved Artificial Agents , 2001, Neural Computation.

[46]  John E. Laird,et al.  Human-Level AI's Killer Application: Interactive Computer Games , 2000, AI Mag..

[47]  R. Pfeifer,et al.  Repeated structure and dissociation of genotypic and phenotypic complexity in artificial ontogeny , 2001 .

[48]  John E. Laird,et al.  Learning procedural knowledge through observation , 2001, K-CAP '01.

[49]  Jordan B. Pollack,et al.  Creating High-Level Components with a Generative Representation for Body-Brain Evolution , 2002, Artificial Life.

[50]  Robert McCartney,et al.  Generating war game strategies using a genetic algorithm , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[51]  Risto Miikkulainen,et al.  Efficient Reinforcement Learning Through Evolving Neural Network Topologies , 2002, GECCO.

[52]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[53]  Benjamin Geisler,et al.  An Empirical Study of Machine Learning Algorithms Applied to Modeling Player Behavior in a "First Person Shooter" Video Game , 2002 .

[54]  Risto Miikkulainen,et al.  Neuroevolution for adaptive teams , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[55]  Risto Miikkulainen,et al.  Active Guidance for a Finless Rocket Using Neuroevolution , 2003, GECCO.

[56]  Risto Miikkulainen,et al.  Robust non-linear control through neuroevolution , 2003 .

[57]  Marcus Gallagher,et al.  Learning to play Pac-Man: an evolutionary, rule-based approach , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[58]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[59]  L. D. Whitley.  Genetic reinforcement learning for neurocontrol problems , 1993, Machine Learning.

[60]  L. D. Whitley,et al.  Genetic Reinforcement Learning for Neurocontrol Problems , 2004, Machine Learning.

[61]  Paul E. Utgoff,et al.  Incremental Induction of Decision Trees , 1989, Machine Learning.

[62]  Risto Miikkulainen,et al.  Evolving Neural Networks to Play Go , 2004, Applied Intelligence.

[63]  Risto Miikkulainen,et al.  Online Interactive Neuro-evolution , 2000, Neural Processing Letters.

[64]  Risto Miikkulainen,et al.  Competitive Coevolution through Evolutionary Complexification , 2011, J. Artif. Intell. Res..

[65]  John Levine,et al.  An evolutionary approach for interactive computer games , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[66]  Jim Esch,et al.  A Self-Learning Evolutionary Chess Program , 2004, Proc. IEEE.

[67]  Sung-Bae Cho,et al.  Evolution of emergent behaviors for shooting game characters in Robocode , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[68]  Sushil J. Louis,et al.  Using a genetic algorithm to tune first-person shooter bots , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[69]  Daryl Essam.  Book Review: Blondie24: Playing at the Edge of AI , 2004, Genetic Programming and Evolvable Machines.

[70]  Hans-Paul Schwefel,et al.  Evolution strategies – A comprehensive introduction , 2002, Natural Computing.

[71]  Risto Miikkulainen,et al.  Efficient Reinforcement Learning through Symbiotic Evolution , 2004 .

[72]  David B. Fogel,et al.  A platform for evolving characters in competitive games , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[73]  Risto Miikkulainen,et al.  Evolving a Roving Eye for Go , 2004, GECCO.

[74]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[75]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[76]  Nicholas J. Radcliffe,et al.  Genetic set recombination and its application to neural network topology optimisation , 1993, Neural Computing & Applications.

[77]  W. Spears.  Speciation Using Tag Bits , 2007 .