Evolving Multimodal Networks for Multitask Games

Intelligent opponent behavior makes video games interesting to human players. Evolutionary computation can discover such behavior, however, it is challenging to evolve behavior that consists of multiple separate tasks. This paper evaluates three ways of meeting this challenge via neuroevolution: 1) multinetwork learns separate controllers for each task, which are then combined manually; 2) multitask evolves separate output units for each task, but shares information within the network's hidden layer; and 3) mode mutation evolves new output modes, and includes a way to arbitrate between them. Whereas the fist two methods require that the task division be known, mode mutation does not. Results in Front/Back Ramming and Predator/Prey games show that each of these methods has different strengths. Multinetwork is good in both domains, taking advantage of the clear division between tasks. Multitask performs well in Front/Back Ramming, in which the relative difficulty of the tasks is even, but poorly in Predator/Prey, in which it is lopsided. Interestingly, mode mutation adapts to this asymmetry and performs well in Predator/Prey. This result demonstrates how a human-specified task division is not always the best. Altogether the results suggest how human knowledge and learning can be combined most effectively to evolve multimodal behavior.

[1]  Risto Miikkulainen,et al.  Automatic feature selection in neuroevolution , 2005, GECCO '05.

[2]  Bernhard Hengst,et al.  Discovering Hierarchy in Reinforcement Learning with HEXQ , 2002, ICML.

[3]  Hussein A. Abbass,et al.  Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[4]  Risto Miikkulainen,et al.  Real-Time Evolution of Neural Networks in the NERO Video Game , 2006, AAAI.

[5]  Georgios N. Yannakakis,et al.  Interactive opponents generate interesting games , 2004 .

[6]  Thomas G. Dietterich.  The MAXQ Method for Hierarchical Reinforcement Learning , 1998, ICML.

[7]  Lothar Thiele,et al.  The Hypervolume Indicator Revisited: On the Design of Pareto-compliant Indicators Via Weighted Integration , 2007, EMO.

[8]  Marco Laumanns,et al.  Performance assessment of multiobjective optimizers: an analysis and review , 2003, IEEE Trans. Evol. Comput..

[9]  Xin Yao,et al.  Neural-Based Learning Classifier Systems , 2008, IEEE Transactions on Knowledge and Data Engineering.

[10]  Astro Teller,et al.  Evolving Team Darwin United , 1998, RoboCup.

[11]  Dana H. Ballard,et al.  Multiple-Goal Reinforcement Learning with Modular Sarsa(0) , 2003, IJCAI.

[12]  浅田 稔,et al.  RoboCup-98 : Robot Soccer World Cup II , 1999 .

[13]  Risto Miikkulainen,et al.  Retaining Learned Behavior During Real-Time Neuroevolution , 2005, AIIDE.

[14]  Stefano Nolfi,et al.  Duplication of Modules Facilitates the Evolution of Functional Specialization , 1999, Artificial Life.

[15]  Risto Miikkulainen,et al.  The role of reward structure, coordination mechanism and net return in the evolution of cooperation , 2011, CIG.

[16]  Lothar Thiele,et al.  A Tutorial on the Performance Assessment of Stochastic Multiobjective Optimizers , 2006 .

[17]  Risto Miikkulainen,et al.  Evolving Stochastic Controller Networks for Intelligent Game Agents , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[18]  Kenneth O. Stanley,et al.  Constraining connectivity to encourage modularity in HyperNEAT , 2011, GECCO '11.

[19]  Risto Miikkulainen,et al.  Active Guidance for a Finless Rocket Using Neuroevolution , 2003, GECCO.

[20]  Risto Miikkulainen,et al.  Constructing complex NPC behavior via multi-objective neuroevolution , 2008, AAAI 2008.

[21]  Manuela M. Veloso,et al.  Multiagent Systems: A Survey from a Machine Learning Perspective , 2000, Auton. Robots.

[22]  Frédéric Gruau,et al.  Automatic Definition of Modular Neural Networks , 1994, Adapt. Behav..

[23]  Jonathan Klein,et al.  breve: a 3D environment for the simulation of decentralized systems and artificial life , 2002 .

[24]  Risto Miikkulainen,et al.  Evolving agent behavior in multiobjective domains using fitness-based shaping , 2010, GECCO '10.

[25]  Alan Fern,et al.  Multi-task reinforcement learning: a hierarchical Bayesian approach , 2007, ICML '07.

[26]  Risto Miikkulainen,et al.  Evolving adaptive neural networks with and without adaptive synapses , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[27]  Tom Ziemke.  On 'Parts' and 'Wholes' of Adaptive Behavior: Functional Modularity and Diachronic Structure in Recurrent Neural Robot Controllers , 2000 .

[28]  Julian Togelius,et al.  Evolution of a subsumption architecture neurocontroller , 2004, J. Intell. Fuzzy Syst..

[29]  Kalyanmoy Deb,et al.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II , 2000, PPSN.

[30]  John Levine,et al.  Improving control through subsumption in the EvoTanks domain , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[31]  Julian Togelius,et al.  Hierarchical controller learning in a First-Person Shooter , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[32]  Xin Yao,et al.  Co-evolutionary modular neural networks for automatic problem decomposition , 2005, 2005 IEEE Congress on Evolutionary Computation.

[33]  Julian Togelius,et al.  Super mario evolution , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[34]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[35]  Jean-Arcady Meyer,et al.  Evolution and Development of Modular Control Architectures for 1D Locomotion in Six-legged Animats , 1998, Connect. Sci..

[36]  Rodney A. Brooks,et al.  A Robust Layered Control Syste For A Mobile Robot , 2022 .

[37]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[38]  Masayuki Yamamura,et al.  Multitask reinforcement learning on the distribution of MDPs , 2003, Proceedings 2003 IEEE International Symposium on Computational Intelligence in Robotics and Automation. Computational Intelligence in Robotics and Automation for the New Millennium (Cat. No.03EX694).

[39]  Peter J. Fleming,et al.  An Overview of Evolutionary Algorithms in Multiobjective Optimization , 1995, Evolutionary Computation.

[40]  Risto Miikkulainen,et al.  Evolving neural networks for strategic decision-making problems , 2009, Neural Networks.

[41]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[42]  Risto Miikkulainen,et al.  Evolving multi-modal behavior in NPCs , 2009, CIG.

[43]  Daniele Loiacono,et al.  Evolving competitive car controllers for racing games with neuroevolution , 2009, GECCO '09.

[44]  Dario Floreano,et al.  Genetic Team Composition and Level of Selection in the Evolution of Cooperation , 2009, IEEE Transactions on Evolutionary Computation.

[45]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..