Evolution and Learning in an Intrinsically Motivated Reinforcement Learning Robot

Studying the role played by evolution and learning in adaptive behavior is a very important topic in artificial life research. This paper investigates the interplay between learning and evolution when agents have to solve several different tasks, as it is the case for real organisms but typically not for artificial agents. Recently, an important thread of research in machine learning and developmental robotics has begun to investigate how agents can solve different tasks by composing general skills acquired on the basis of internal motivations. This work presents a hierarchical, neural-network, actor-critic architecture designed for implementing this kind of intrinsically motivated reinforcement learning in real robots. We compare the results of several experiments in which the various components of the architecture are either trained during lifetime or evolved through a genetic algorithm. The most important results show that systems using both evolution and learning outperform systems using either one of the two, and that, among the former, systems evolving internal reinforcers for learning building-block skills have a higher evolvability than those directly evolving the related behaviors.

[1]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[2]  Jürgen Schmidhuber,et al.  A possibility for implementing curiosity and boredom in model-building neural controllers , 1991 .

[3]  Domenico Parisi,et al.  Evolving Modular Architectures for Neural Networks , 2000, NCPW.

[4]  Isaac Meilijson,et al.  Evolution of reinforcement learning in foraging bees: a simple explanation for risk averse behavior , 2002, Neurocomputing.

[5]  Robert M. French,et al.  Connectionist Models of Learning, Development and Evolution , 2001, Perspectives in Neural Computing.

[6]  Stefano Nolfi,et al.  Learning and Evolution , 1999, Auton. Robots.

[7]  Douglas S. Blank,et al.  An Emergent Framework For Self-Motivation In Developmental Robotics , 2004 .

[8]  Kenji Doya,et al.  Evolution of meta-parameters in reinforcement learning algorithm , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[9]  G. Baldassarre,et al.  Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot , 2007, 2007 IEEE 6th International Conference on Development and Learning.

[10]  W. Schultz.  Getting Formal with Dopamine and Reward , 2002, Neuron.

[11]  Jean-Arcady Meyer,et al.  From Animals to Animats: Proceedings of The First International Conference on Simulation of Adaptive Behavior (Complex Adaptive Systems) , 1990 .

[12]  Andrew G. Barto,et al.  Intrinsically Motivated Reinforcement Learning: A Promising Framework for Developmental Robot Learning , 2005 .

[13]  Gianluca Baldassarre,et al.  A modular neural-network model of the basal ganglia’s role in learning and selecting motor behaviours , 2002, Cognitive Systems Research.

[14]  David H. Ackley,et al.  Interactions between learning and evolution , 1991 .

[15]  John Maynard Smith,et al.  When learning guides evolution , 1987, Nature.

[16]  Michael A. Arbib,et al.  The handbook of brain theory and neural networks , 1995, A Bradford book.

[17]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[18]  Jeffrey L. Elman,et al.  Learning and Evolution in Neural Networks , 1994, Adapt. Behav..

[19]  Dario Floreano,et al.  Evolution of Adaptive Synapses: Robots with Fast Adaptive Behavior in New Environments , 2001, Evolutionary Computation.

[20]  Xiao Huang,et al.  Novelty and Reinforcement Learning in the Value System of Developmental Robots , 2002 .

[21]  Pierre-Yves Oudeyer,et al.  Intrinsic Motivation Systems for Autonomous Mental Development , 2007, IEEE Transactions on Evolutionary Computation.

[22]  Joel L. Davis,et al.  Adaptive Critics and the Basal Ganglia , 1995 .

[23]  Stefano Nolfi,et al.  Evolutionary Robotics: Exploiting the Full Power of Self-organization , 1998, Connect. Sci..

[24]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[25]  Charles E. Taylor,et al.  Artificial Life II , 1991 .

[26]  Geoffrey E. Hinton,et al.  How Learning Can Guide Evolution , 1996, Complex Syst..

[27]  R. W. White.  Motivation reconsidered: the concept of competence. , 1959, Psychological review.

[28]  Nuttapong Chentanez,et al.  Intrinsically Motivated Learning of Hierarchical Collections of Skills , 2004 .

[29]  G. Baldassarre,et al.  A neural-network reinforcement-learning model of domestic chicks that learn to localize the centre of closed arenas , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[30]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .