Multi-agent Behavior-Based Policy Transfer

A key objective of transfer learning is to improve and speed-up learning on a target task after training on a different, but related, source task. This study presents a neuro-evolution method that transfers evolved policies within multi-agent tasks of varying degrees of complexity. The method incorporates behavioral diversity (novelty) search as a means to boost the task performance of transferred policies (multi-agent behaviors). Results indicate that transferred evolved multi-agent behaviors are significantly improved in more complex tasks when adapted using behavioral diversity. Comparatively, behaviors that do not use behavioral diversity to further adapt transferred behaviors, perform relatively poorly in terms of adaptation times and quality of solutions in target tasks. Also, in support of previous work, both policy transfer methods (with and without behavioral diversity adaptation), out-perform behaviors evolved in target tasks without transfer learning.

[1]  Jude Shavlik,et al.  Chapter 11 Transfer Learning , 2009 .

[2]  Kenneth O. Stanley,et al.  A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks , 2009, Artificial Life.

[3]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[4]  Stéphane Doncieux,et al.  Encouraging Behavioral Diversity in Evolutionary Robotics: An Empirical Study , 2012, Evolutionary Computation.

[5]  Shimon Whiteson,et al.  Transfer Learning for Policy Search Methods , 2006 .

[6]  Anders Lyhne Christensen,et al.  Avoiding convergence in cooperative coevolution with novelty search , 2014, AAMAS.

[7]  Jonas Degrave,et al.  Transfer learning of gaits on a quadrupedal robot , 2015, Adapt. Behav..

[8]  Kenneth O. Stanley,et al.  Compositional Pattern Producing Networks : A Novel Abstraction of Development , 2007 .

[9]  Kenneth O. Stanley,et al.  Revising the evolutionary computation abstraction: minimal criteria novelty search , 2010, GECCO '10.

[10]  William H. Press,et al.  Numerical recipes , 1990 .

[11]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[12]  Jeff Clune,et al.  Novelty search creates robots with general skills for exploration , 2014, GECCO.

[13]  Kenneth O. Stanley.  A Hypercube-Based Indirect Encoding for Evolving Large-Scale Neural Networks , 2009 .

[14]  Kenneth O. Stanley,et al.  A Case Study on the Critical Role of Geometric Regularity in Machine Learning , 2008, AAAI.

[15]  Peter Stone,et al.  Transfer Learning via Inter-Task Mappings for Temporal Difference Learning , 2007, J. Mach. Learn. Res..

[16]  Amiram Moshaiov,et al.  Family bootstrapping: A genetic transfer learning approach for onsetting the evolution for a set of related robotic tasks , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[17]  Anders Lyhne Christensen,et al.  Devising Effective Novelty Search Algorithms: A Comprehensive Empirical Study , 2015, GECCO.

[18]  Anders Lyhne Christensen,et al.  Progressive Minimal Criteria Novelty Search , 2012, IBERAMIA.

[19]  Ioannis P. Vlahavas,et al.  Transfer Learning in Multi-Agent Reinforcement Learning Domains , 2011, EWRL.

[20]  Haitham Bou-Ammar,et al.  Reinforcement learning transfer via sparse coding , 2012, AAMAS.

[21]  Stéphane Doncieux.  Knowledge Extraction from Learning Traces in Continuous Domains , 2014, AAAI Fall Symposia.

[22]  Kurt Driessens,et al.  Transfer Learning in Reinforcement Learning Problems Through Partial Policy Recycling , 2007, ECML.

[23]  Sebastian Risi,et al.  Confronting the challenge of learning a flexible neural controller for a diversity of morphologies , 2013, GECCO '13.

[24]  Peter Stone,et al.  Reinforcement Learning for RoboCup Soccer Keepaway , 2005, Adapt. Behav..

[25]  Kagan Tumer,et al.  Policy transfer in mobile robots using neuro-evolutionary navigation , 2012, GECCO '12.

[26]  Julian Togelius,et al.  Constrained Novelty Search: A Study on Game Content Generation , 2015, Evolutionary Computation.

[27]  Kenneth O. Stanley,et al.  Abandoning Objectives: Evolution Through the Search for Novelty Alone , 2011, Evolutionary Computation.

[28]  Kenneth O. Stanley,et al.  Efficiently evolving programs through the search for novelty , 2010, GECCO '10.

[29]  Peter Stone,et al.  Keepaway Soccer: From Machine Learning Testbed to Benchmark , 2005, RoboCup.

[30]  L. Buşoniu.  Evolutionary function approximation for reinforcement learning , 2006 .

[31]  Kenneth O. Stanley,et al.  Evolving Static Representations for Task Transfer , 2010, J. Mach. Learn. Res..

[32]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[33]  Paulo Urbano,et al.  Improving Grammatical Evolution in Santa Fe Trail using Novelty Search , 2013, ECAL.

[34]  Kenneth O. Stanley,et al.  Scalable multiagent learning through indirect encoding of policy geometry , 2013, Evol. Intell..

[35]  Risto Miikkulainen,et al.  Transfer of evolved pattern-based heuristics in games , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[36]  Faustino J. Gomez,et al.  Novelty-based restarts for evolution strategies , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[37]  Anders Lyhne Christensen,et al.  Generic behaviour similarity measures for evolutionary swarm robotics , 2013, GECCO '13.

[38]  A. Ghasemi,et al.  Normality Tests for Statistical Analysis: A Guide for Non-Statisticians , 2012, International journal of endocrinology and metabolism.