Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot

Intrinsically motivated reinforcement learning (IMRL) has been proposed as a framework within which agents exploit "internal reinforcement" to acquire general-purpose building-block behaviors ("skills") which can be later combined for solving several specific tasks. The architectures so far proposed within this framework are limited in that: (1) they use hardwired "salient events" to form and train skills, and this limits agents' autonomy; (2) they are applicable only to problems with abstract states and actions, as grid-world problems. This paper proposes solutions to these problems in the form of a hierarchical reinforcement-learning architecture that: (1) exploits evolutionary robotics techniques so to allow the system to autonomously discover "salient events"; (2) uses neural networks so to allow the system to cope with continuous states and noisy environments. The viability of the proposed approach is demonstrated with a simulated robotic scenario.

[1]  R. W. White.  Motivation reconsidered: the concept of competence. , 1959, Psychological review.

[2]  Stewart W. Wilson,et al.  A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers , 1991 .

[3]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[4]  Stefano Nolfi,et al.  Evolutionary Robotics: Exploiting the Full Power of Self-organization , 1998, Connect. Sci..

[5]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[6]  Stefano Nolfi,et al.  Evolutionary Robotics: The Biology, Intelligence, and Technology of Self-Organizing Machines , 2000 .

[7]  D. Floreano,et al.  Evolutionary Robotics: The Biology,Intelligence,and Technology , 2000 .

[8]  James L. McClelland,et al.  Autonomous Mental Development by Robots and Animals , 2001, Science.

[9]  Xiao Huang,et al.  Novelty and Reinforcement Learning in the Value System of Developmental Robots , 2002 .

[10]  W. Schultz.  Getting Formal with Dopamine and Reward , 2002, Neuron.

[11]  P. Dayan,et al.  Reward, Motivation, and Reinforcement Learning , 2002, Neuron.

[12]  Gianluca Baldassarre,et al.  A modular neural-network model of the basal ganglia’s role in learning and selecting motor behaviours , 2002, Cognitive Systems Research.

[13]  Pierre-Yves Oudeyer,et al.  Motivational principles for visual know-how development , 2003 .

[14]  Takashi Gomi,et al.  Book Review: Evolutionary Robotics: the Biology, Intelligence, and Technology of Self-Organizing Machines , 2003, Genetic Programming and Evolvable Machines.

[15]  Nuttapong Chentanez,et al.  Intrinsically Motivated Learning of Hierarchical Collections of Skills , 2004 .

[16]  Douglas S. Blank,et al.  An Emergent Framework For Self-Motivation In Developmental Robotics , 2004 .

[17]  Andrew G. Barto,et al.  Intrinsically Motivated Reinforcement Learning: A Promising Framework for Developmental Robot Learning , 2005 .

[18]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[19]  Andrew G. Barto,et al.  An intrinsic reward mechanism for efficient exploration , 2006, ICML.

[20]  P. Redgrave,et al.  The short-latency dopamine signal: a role in discovering novel actions? , 2006, Nature Reviews Neuroscience.

[21]  G. Baldassarre,et al.  A neural-network reinforcement-learning model of domestic chicks that learn to localize the centre of closed arenas , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[22]  Wolfram Schultz,et al.  Reward , 2019, HR for Creative Companies.

[23]  Marco Mirolli,et al.  Evolution and Learning in an Intrinsically Motivated Reinforcement Learning Robot , 2007, ECAL.

[24]  J. Croft.  Conflict , 2007, The Evolution of Social Behaviour.

[25]  Pierre-Yves Oudeyer,et al.  Intrinsic Motivation Systems for Autonomous Mental Development , 2007, IEEE Transactions on Evolutionary Computation.

[26]  Richard S. Sutton,et al.  Reinforcement Learning , 1992, Handbook of Machine Learning.