Balancing selection pressures, multiple objectives, and neural modularity to coevolve cooperative agent behavior

Previous research using evolutionary computation in Multi-Agent Systems indicates that assigning fitness based on team vs. individual behavior has a strong impact on the ability of evolved teams of artificial agents to exhibit teamwork in challenging tasks. Such research made use of single-objective evolution, but when multiobjective evolution is used, populations can be subject to individual-level objectives, team-level objectives, or combinations of the two. This paper explores the performance of cooperatively coevolved teams of agents controlled by artificial neural networks subject to these types of objectives. Because of the tension between individual and team behaviors, multiple modes of behavior can be useful, so the effect of modular neural networks is also explored. Results demonstrate that fitness rewarding individual behavior is superior to fitness rewarding team behavior, despite being applied to a cooperative task. However, networks with multiple modules can discover intelligent behavior, regardless of which type of objectives are used.

[1]  Risto Miikkulainen,et al.  Evolving Multimodal Networks for Multitask Games , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[2]  Dario Floreano,et al.  Genetic Team Composition and Level of Selection in the Evolution of Cooperation , 2009, IEEE Transactions on Evolutionary Computation.

[3]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[4]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[5]  Geoff S. Nitschke,et al.  Approaches to dynamic team sizes , 2013, 2013 IEEE Workshop on Robotic Intelligence in Informationally Structured Space (RiiSS).

[6]  Jean-Baptiste Mouret,et al.  Does Aligning Phenotypic and Genotypic Modularity Improve the Evolution of Neural Networks? , 2016, GECCO.

[7]  Dario Floreano,et al.  Evolution of Altruistic Robots , 2008, WCCI.

[8]  Martijn C. Schut,et al.  Evolving team behaviors with specialization , 2012, Genetic Programming and Evolvable Machines.

[9]  Risto Miikkulainen,et al.  Constructing complex NPC behavior via multi-objective neuroevolution , 2008, AAAI 2008.

[10]  Manuela M. Veloso,et al.  Multiagent Systems: A Survey from a Machine Learning Perspective , 2000, Auton. Robots.

[11]  Julian Togelius,et al.  Hierarchical controller learning in a First-Person Shooter , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[12]  Kagan Tumer,et al.  Unifying temporal and structural credit assignment problems , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..

[13]  E. Leigh,et al.  The group selection controversy , 2010 .

[14]  Kenneth O. Stanley,et al.  Constraining connectivity to encourage modularity in HyperNEAT , 2011, GECCO '11.

[15]  Kenneth A. De Jong,et al.  Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents , 2000, Evolutionary Computation.

[16]  Risto Miikkulainen,et al.  Constructing competitive and cooperative agent behavior using coevolution , 2010, CIG.

[17]  Richard A. Watson,et al.  Reducing Local Optima in Single-Objective Problems by Multi-objectivization , 2001, EMO.

[18]  Risto Miikkulainen,et al.  Coevolution of Role-Based Cooperation in Multiagent Systems , 2009, IEEE Transactions on Autonomous Mental Development.

[19]  Risto Miikkulainen,et al.  Incremental Evolution of Complex General Behavior , 1997, Adapt. Behav..

[20]  Sandip Sen,et al.  Evolving Beharioral Strategies in Predators and Prey , 1995, Adaption and Learning in Multi-Agent Systems.

[21]  Risto Miikkulainen,et al.  The role of reward structure, coordination mechanism and net return in the evolution of cooperation , 2011, CIG.

[22]  Annie S. Wu,et al.  Multi-agent role allocation: issues, approaches, and multiple perspectives , 2011, Autonomous Agents and Multi-Agent Systems.

[23]  Risto Miikkulainen,et al.  Discovering Multimodal Behavior in Ms. Pac-Man Through Evolution of Modular Neural Networks , 2016, IEEE Transactions on Computational Intelligence and AI in Games.

[24]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[25]  U. Alon,et al.  Spontaneous evolution of modularity and network motifs. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[26]  Hod Lipson,et al.  The evolutionary origins of modularity , 2012, Proceedings of the Royal Society B: Biological Sciences.