Relational Macros for Transfer in Reinforcement Learning

We describe an application of inductive logic programming to transfer learning. Transfer learning is the use of knowledge learned in a source task to improve learning in a related target task. The tasks we work with are in reinforcement-learning domains. Our approach transfers relational macros, which are finite-state machines in which the transition conditions and the node actions are represented by first-order logical clauses. We use inductive logic programming to learn a macro that characterizes successful behavior in the source task, and then use the macro for decision-making in the early learning stages of the target task. Through experiments in the RoboCup simulated soccer domain, we show that Relational Macro Transfer via Demonstration (RMT-D) from a source task can provide a substantial head start in the target task.

[1]  S. Seshu,et al.  Introduction to the theory of finite-state machines , 1963 .

[2]  Ben J. A. Kröse,et al.  Learning from delayed rewards , 1995, Robotics Auton. Syst..

[3]  Ian Frank,et al.  Soccer Server: A Tool for Research on Multiagent Systems , 1998, Appl. Artif. Intell..

[4]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[5]  Doina Precup,et al.  Using Options for Knowledge Transfer in Reinforcement Learning , 1999 .

[6]  Thomas G. Dietterich.  Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition , 1999, J. Artif. Intell. Res..

[7]  Peter Stone,et al.  Scaling Reinforcement Learning toward RoboCup Soccer , 2001, ICML.

[8]  Saso Dzeroski,et al.  Integrating Guidance into Relational Reinforcement Learning , 2004, Machine Learning.

[9]  Robert Givan,et al.  Relational Reinforcement Learning: An Overview , 2004, ICML 2004.

[10]  Peter Stone,et al.  Value Functions for RL-Based Behavior Transfer: A Comparative Study , 2005, AAAI.

[11]  Richard Maclin,et al.  Knowledge-Based Support-Vector Regression for Reinforcement Learning , 2005 .

[12]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[13]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[14]  Jude W. Shavlik,et al.  Using Advice to Transfer Knowledge Acquired in One Reinforcement Learning Task to Another , 2005, ECML.

[15]  Jude W. Shavlik,et al.  Giving Advice about Preferred Actions to Reinforcement Learners Via Knowledge-Based Kernel Regression , 2005, AAAI.

[16]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[17]  Pat Langley,et al.  Relational temporal difference learning , 2006, ICML.

[18]  Lisa A. Torrey.  Relational Skill Transfer via Advice Taking , 2006 .

[19]  Fernando Fernández,et al.  Policy Reuse for Transfer Learning Across Tasks with Different State and Action Spaces , 2006 .

[20]  D. Stracuzzi.  Transfer of Knowledge Structures with Relational Temporal Difference Learning , 2006 .

[21]  Vishal Soni,et al.  Using Homomorphisms to Transfer Options across Continuous Reinforcement Learning Domains , 2006, AAAI.

[22]  Jan Ramon,et al.  Transfer learning for reinforcement learning through goal and policy parametrization , 2006, ICML 2006.

[23]  Jude W. Shavlik,et al.  Skill Acquisition Via Transfer Learning and Advice Taking , 2006, ECML.

[24]  Neil D. Lawrence,et al.  Missing Data in Kernel PCA , 2006, ECML.

[25]  Maurice Bruynooghe,et al.  Learning Relational Options for Inductive Transfer in Relational Reinforcement Learning , 2007, ILP.

[26]  Peter Stone,et al.  Cross-domain transfer for reinforcement learning , 2007, ICML '07.