Transfer Learning via Advice Taking

The goal of transfer learning is to speed up learning in a new task by transferring knowledge from one or more related source tasks. We describe a transfer method in which a reinforcement learner analyzes its experience in the source task and learns rules to use as advice in the target task. The rules, which are learned via inductive logic programming, describe the conditions under which an action is successful in the source task. The advice-taking algorithm used in the target task allows a reinforcement learner to benefit from rules even if they are imperfect. A human-provided mapping describes the alignment between the source and target tasks, and may also include advice about the differences between them. Using three tasks in the RoboCup simulated soccer domain, we demonstrate that this transfer method can speed up reinforcement learning substantially.

[1]  Andrew G. Barto,et al.  Autonomous shaping: knowledge transfer in reinforcement learning , 2006, ICML.

[2]  Eric E. Bloedorn,et al.  Multistrategy constructive induction , 1997 .

[3]  Richard Maclin,et al.  Knowledge-Based Support-Vector Regression for Reinforcement Learning , 2005 .

[4]  Peter Stone,et al.  Scaling Reinforcement Learning toward RoboCup Soccer , 2001, ICML.

[5]  Alan Fern,et al.  Multi-task reinforcement learning: a hierarchical Bayesian approach , 2007, ICML '07.

[6]  Andrea Bonarini,et al.  Improving Batch Reinforcement Learning Performance through Transfer of Samples , 2008, STAIRS.

[7]  Ryszard S. Michalski,et al.  Toward a unified theory of learning: multistrategy task-adaptive learning , 1993 .

[8]  Ryszard S. Michalski,et al.  A Theory and Methodology of Inductive Learning , 1983, Artificial Intelligence.

[9]  R. Michalski,et al.  Multistrategy Constructive Induction: AQ17-MCI , 1993 .

[10]  Satinder Singh.  Transfer of Learning by Composing Solutions of Elemental Sequential Tasks , 1992, Mach. Learn..

[11]  Jude W. Shavlik,et al.  Knowledge-Based Kernel Approximation , 2004, J. Mach. Learn. Res..

[12]  Doina Precup,et al.  Using Options for Knowledge Transfer in Reinforcement Learning , 1999 .

[13]  Jude W. Shavlik,et al.  A Simple and Effective Method for Incorporating Advice into Kernel Methods , 2006, AAAI.

[14]  Michael G. Madden,et al.  Transfer of Experience Between Reinforcement Learning Environments with Progressive Difficulty , 2004, Artificial Intelligence Review.

[15]  Thomas J. Walsh.  Transferring State Abstractions Between MDPs , 2006 .

[16]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[17]  Thomas G. Dietterich,et al.  Automatic discovery and transfer of MAXQ hierarchies , 2008, ICML '08.

[18]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[19]  Shimon Whiteson,et al.  Transfer Learning for Policy Search Methods , 2006 .

[20]  Craig Boutilier,et al.  Implicit Imitation in Multiagent Reinforcement Learning , 1999, ICML.

[21]  Ian Frank,et al.  Soccer Server: A Tool for Research on Multiagent Systems , 1998, Appl. Artif. Intell..

[22]  Peter Stone,et al.  Value Functions for RL-Based Behavior Transfer: A Comparative Study , 2005, AAAI.

[23]  Tanaka Fumihide,et al.  Multitask Reinforcement Learning on the Distribution of MDPs , 2003 .

[24]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[25]  Neil D. Lawrence,et al.  Missing Data in Kernel PCA , 2006, ECML.

[26]  David C. Wilkins,et al.  Readings in Knowledge Acquisition and Learning: Automating the Construction and Improvement of Expert Systems , 1992 .

[27]  Jude W. Shavlik,et al.  Using Advice to Transfer Knowledge Acquired in One Reinforcement Learning Task to Another , 2005, ECML.

[28]  Ashwin Ram,et al.  Transfer Learning in Real-Time Strategy Games Using Hybrid CBR/RL , 2007, IJCAI.

[29]  Thomas G. Dietterich.  Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition , 1999, J. Artif. Intell. Res..

[30]  Peter Stone,et al.  Transferring Instances for Model-Based Reinforcement Learning , 2008, ECML/PKDD.

[31]  Peter A. Flach,et al.  Evaluation Measures for Multi-class Subgroup Discovery , 2009, ECML/PKDD.

[32]  Manfred Huber,et al.  Effective Control Knowledge Transfer through Learning Skill and Representation Hierarchies , 2007, IJCAI.

[33]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[34]  Jude W. Shavlik,et al.  Skill Acquisition Via Transfer Learning and Advice Taking , 2006, ECML.

[35]  Manuela M. Veloso,et al.  Probabilistic policy reuse in a reinforcement learning agent , 2006, AAMAS '06.

[36]  Alexander A. Sherstov,et al.  Action-Space Knowledge Transfer in MDP ’ s : Formalism , Suboptimality Bounds , and Algorithms ? , 2005 .

[37]  Jude W. Shavlik,et al.  Giving Advice about Preferred Actions to Reinforcement Learners Via Knowledge-Based Kernel Regression , 2005, AAAI.

[38]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[39]  Peter Stone,et al.  Cross-domain transfer for reinforcement learning , 2007, ICML '07.

[40]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[41]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[42]  Maurice Bruynooghe,et al.  Learning Relational Options for Inductive Transfer in Relational Reinforcement Learning , 2007, ILP.