Novelty-Driven Cooperative Coevolution

Cooperative coevolutionary algorithms (CCEAs) rely on multiple coevolving populations for the evolution of solutions composed of coadapted components. CCEAs enable, for instance, the evolution of cooperative multiagent systems composed of heterogeneous agents, where each agent is modelled as a component of the solution. Previous works have, however, shown that CCEAs are biased toward stability: the evolutionary process tends to converge prematurely to stable states instead of (near-)optimal solutions. In this study, we show how novelty search can be used to avoid the counterproductive attraction to stable states in coevolution. Novelty search is an evolutionary technique that drives evolution toward behavioural novelty and diversity rather than exclusively pursuing a static objective. We evaluate three novelty-based approaches that rely on, respectively (1) the novelty of the team as a whole, (2) the novelty of the agents’ individual behaviour, and (3) the combination of the two. We compare the proposed approaches with traditional fitness-driven cooperative coevolution in three simulated multirobot tasks. Our results show that team-level novelty scoring is the most effective approach, significantly outperforming fitness-driven coevolution at multiple levels. Novelty-driven cooperative coevolution can substantially increase the potential of CCEAs while maintaining a computational complexity that scales well with the number of populations.

[1]  Stéphane Doncieux,et al.  Encouraging Behavioral Diversity in Evolutionary Robotics: An Empirical Study , 2012, Evolutionary Computation.

[2]  Anders Lyhne Christensen,et al.  Generic behaviour similarity measures for evolutionary swarm robotics , 2013, GECCO '13.

[3]  Anders Lyhne Christensen,et al.  Cooperative Coevolution of Morphologically Heterogeneous Robots , 2015, ECAL.

[4]  Anders Lyhne Christensen,et al.  Avoiding convergence in cooperative coevolution with novelty search , 2014, AAMAS.

[5]  Kenneth A. De Jong,et al.  The dynamics of the best individuals in co-evolution , 2006, Natural Computing.

[6]  Kenneth A. De Jong,et al.  Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents , 2000, Evolutionary Computation.

[7]  Liviu Panait,et al.  Theoretical Convergence Guarantees for Cooperative Coevolutionary Algorithms , 2010, Evolutionary Computation.

[8]  Kenneth O. Stanley,et al.  Exploiting Open-Endedness to Solve Problems Through the Search for Novelty , 2008, ALIFE.

[9]  Jeffrey K. Bassett,et al.  An Analysis of Cooperative Coevolutionary Algorithms A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at George Mason University , 2003 .

[10]  Anders Lyhne Christensen,et al.  Evolution of swarm robotics systems with novelty search , 2013, Swarm Intelligence.

[11]  Michael I. Jordan.  Serial Order: A Parallel Distributed Processing Approach , 1997 .

[12]  Thomas Jansen,et al.  Exploring the Explorative Advantage of the Cooperative Coevolutionary (1+1) EA , 2003, GECCO.

[13]  Charles E. Hughes,et al.  Evolving plastic neural networks with novelty search , 2010, Adapt. Behav..

[14]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[15]  Anders Lyhne Christensen,et al.  Cooperative Coevolution of Partially Heterogeneous Multiagent Systems , 2015, AAMAS.

[16]  Stéphane Doncieux,et al.  Behavioral diversity with multiple behavioral distances , 2013, 2013 IEEE Congress on Evolutionary Computation.

[17]  Faustino J. Gomez,et al.  When Novelty Is Not Enough , 2011, EvoApplications.

[18]  Martijn C. Schut,et al.  Evolving team behaviors with specialization , 2012, Genetic Programming and Evolvable Machines.

[19]  Martijn C. Schut,et al.  Collective neuro-evolution for evolving specialized sensor resolutions in a multi-rover task , 2010, Evol. Intell..

[20]  Risto Miikkulainen,et al.  Incremental Evolution of Complex General Behavior , 1997, Adapt. Behav..

[21]  R. Paul Wiegand,et al.  Biasing Coevolutionary Search for Optimal Multiagent Behaviors , 2006, IEEE Transactions on Evolutionary Computation.

[22]  Risto Miikkulainen,et al.  Effective diversity maintenance in deceptive domains , 2013, GECCO '13.

[23]  Sean Luke,et al.  Archive-based cooperative coevolutionary algorithms , 2006, GECCO '06.

[24]  Leonardo Trujillo,et al.  Searching for novel clustering programs , 2013, GECCO '13.

[25]  Kenneth O. Stanley,et al.  Abandoning Objectives: Evolution Through the Search for Novelty Alone , 2011, Evolutionary Computation.

[26]  Stéphane Doncieux,et al.  Beyond black-box optimization: a review of selective pressures for evolutionary robotics , 2014, Evol. Intell..

[27]  Risto Miikkulainen,et al.  Constructing competitive and cooperative agent behavior using coevolution , 2010, CIG.

[28]  Leonardo Trujillo,et al.  Searching for Novel Classifiers , 2013, EuroGP.

[29]  R. Paul Wiegand,et al.  Robustness in cooperative coevolution , 2006, GECCO '06.

[30]  Alan C. Schultz,et al.  Heterogeneity in the Coevolved Behaviors of Mobile Robots: The Emergence of Specialists , 2001, IJCAI.

[31]  Sean Luke,et al.  Cooperative Multi-Agent Learning: The State of the Art , 2005, Autonomous Agents and Multi-Agent Systems.

[32]  Risto Miikkulainen,et al.  Coevolution of Role-Based Cooperation in Multiagent Systems , 2009, IEEE Transactions on Autonomous Mental Development.

[33]  Sean Luke,et al.  MASON: A Multiagent Simulation Environment , 2005, Simul..

[34]  Julian Togelius,et al.  Constrained Novelty Search: A Study on Game Content Generation , 2015, Evolutionary Computation.

[35]  Kenneth A. De Jong,et al.  A Dynamical Systems Analysis of Collaboration Methods in Cooperative Co-evolution , 2005, AAAI Fall Symposium: Coevolutionary and Coadaptive Systems.

[36]  R. K. Ursem.  Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[37]  Martijn C. Schut,et al.  Evolving behavioral specialization in robot teams to solve a collective construction task , 2012, Swarm Evol. Comput..

[38]  Stefano Nolfi,et al.  Co-evolving predator and prey robots , 2012, Adapt. Behav..

[39]  R. Paul Wiegand,et al.  An empirical analysis of collaboration methods in cooperative coevolutionary algorithms , 2001 .

[40]  Kenneth O. Stanley,et al.  Revising the evolutionary computation abstraction: minimal criteria novelty search , 2010, GECCO '10.

[41]  Stéphane Doncieux,et al.  Behavioral diversity measures for Evolutionary Robotics , 2010, IEEE Congress on Evolutionary Computation.

[42]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[43]  L. Darrell Whitley,et al.  Fundamental Principles of Deception in Genetic Search , 1990, FOGA.

[44]  Anders Lyhne Christensen,et al.  Novelty Search in Competitive Coevolution , 2014, PPSN.

[45]  Edwin D. de Jong,et al.  Coevolutionary Principles , 2012, Handbook of Natural Computing.

[46]  R. Paul Wiegand,et al.  A Visual Demonstration of Convergence Properties of Cooperative Coevolution , 2004, PPSN.

[47]  Jean-Baptiste Mouret.  Novelty-Based Multiobjectivization , 2011 .

[48]  Anders Lyhne Christensen,et al.  Systematic Derivation of Behaviour Characterisations in Evolutionary Robotics , 2014, ALIFE.

[49]  Sean Luke,et al.  Time-dependent Collaboration Schemes for Cooperative Coevolutionary Algorithms , 2005, AAAI Fall Symposium: Coevolutionary and Coadaptive Systems.

[50]  Shimon Whiteson,et al.  Critical factors in the performance of novelty search , 2011, GECCO '11.

[51]  Anders Lyhne Christensen,et al.  PMCNS: Using a Progressively Stricter Fitness Criterion to Guide Novelty Search , 2014, Int. J. Nat. Comput. Res..

[52]  Kenneth O. Stanley,et al.  Evolving a diversity of virtual creatures through novelty search and local competition , 2011, GECCO '11.

[53]  John W. Sammon,et al.  A Nonlinear Mapping for Data Structure Analysis , 1969, IEEE Transactions on Computers.

[54]  K.A. De Jong,et al.  Analyzing cooperative coevolution with evolutionary game theory , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[55]  Anders Lyhne Christensen,et al.  Devising Effective Novelty Search Algorithms: A Comprehensive Empirical Study , 2015, GECCO.