Graph based skill acquisition and transfer Learning for continuous reinforcement learning domains

Introducing connectivity graph to model agent behavior and environment dynamics.A graph based approach for automatic skill acquisition.A skill based transfer learning method in continuous reinforcement learning domain. Since reinforcement learning algorithms suffer from the curse of dimensionality in continuous domains, generalization is the most challenging issue in this area. Both skill acquisition and transfer learning are successful techniques to overcome such problem that result in big improvements in agent learning performance. In this paper, we propose a novel graph based skill acquisition method, named GSL, and a skill based transfer learning framework, named STL. GSL discovers skills as high-level knowledge using community detection from connectivity graph, a model to capture not only the agents experience but also the environments dynamics. STL incorporates skills previously learned from source task to speed up learning on a new target task. The experimental results indicate the effectiveness of the proposed methods in dealing with continuous reinforcement learning problems. Display Omitted

[1]  Alessandro Lazaric,et al.  Transfer from Multiple MDPs , 2011, NIPS.

[2]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[3]  Andrew G. Barto,et al.  Skill Characterization Based on Betweenness , 2008, NIPS.

[4]  Stewart W. Wilson Classifier Fitness Based on Accuracy , 1995, Evolutionary Computation.

[5]  Scott Kuindersma,et al.  CST: Constructing Skill Trees by Demonstration , 2011, ICML 2011.

[6]  Hamid Beigy,et al.  A novel graphical approach to automatic abstraction in reinforcement learning , 2013, Robotics Auton. Syst..

[7]  George Konidaris,et al.  Value Function Approximation in Reinforcement Learning Using the Fourier Basis , 2011, AAAI.

[8]  Eran Dayan,et al.  Stochastic reinforcement benefits skill acquisition , 2014, Learning & memory.

[9]  Jieping Ye,et al.  Deep Model Based Transfer and Multi-Task Learning for Biological Image Analysis , 2015, IEEE Transactions on Big Data.

[10]  Simon Coupland,et al.  Fuzzy Transfer Learning: Methodology and application , 2015, Inf. Sci..

[11]  Fan Chung Graham,et al.  A Random Graph Model for Power Law Graphs , 2001, Exp. Math..

[12]  Parham Moradi,et al.  Automatic Skill Acquisition in Reinforcement Learning Agents Using Connection Bridge Centrality , 2010, FGIT-FGCN.

[13]  Ludvig Bohlin,et al.  Community detection and visualization of networks with the map equation framework , 2014 .

[14]  Scott Kuindersma,et al.  Robot learning from demonstration by constructing skill trees , 2012, Int. J. Robotics Res..

[15]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[16]  Sridhar Mahadevan,et al.  Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes , 2007, J. Mach. Learn. Res..

[17]  Alessandro Lazaric,et al.  Transfer in Reinforcement Learning: A Framework and a Survey , 2012, Reinforcement Learning.

[18]  Andrew G. Barto,et al.  Adaptive Step-Size for Online Temporal Difference Learning , 2012, AAAI.

[19]  Manfred Huber,et al.  Effective Control Knowledge Transfer through Learning Skill and Representation Hierarchies , 2007, IJCAI.

[20]  Xiao Li,et al.  Multi-source transfer learning based on label shared subspace , 2015, Pattern Recognit. Lett..

[21]  Jianmin Wang,et al.  Transfer Learning with Graph Co-Regularization , 2012, IEEE Transactions on Knowledge and Data Engineering.

[22]  M. Friedman The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance , 1937 .

[23]  Peter Stone,et al.  Transfer Learning for Reinforcement Learning Domains: A Survey , 2009, J. Mach. Learn. Res..

[24]  Andrew G. Barto,et al.  Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining , 2009, NIPS.

[25]  F. Wilcoxon Individual Comparisons by Ranking Methods , 1945 .

[26]  Scott Kuindersma,et al.  Constructing Skill Trees for Reinforcement Learning Agents from Demonstration Trajectories , 2010, NIPS.

[27]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[28]  Alireza Khadivi,et al.  Automatic skill acquisition in reinforcement learning using graph centrality measures , 2012, Intell. Data Anal..

[29]  Peter Stone,et al.  An Introduction to Intertask Transfer for Reinforcement Learning , 2011, AI Mag..

[30]  Andrea Bonarini,et al.  Transfer of samples in batch reinforcement learning , 2008, ICML '08.

[31]  Gary L. Miller,et al.  Separators for sphere-packings and nearest neighbor graphs , 1997, JACM.

[32]  Jean-Loup Guillaume,et al.  Fast unfolding of communities in large networks , 2008, 0803.0476.