Transfer learning in multi-agent systems through parallel transfer

Transfer Learning(TL) has been shown to signicantly accelerate the convergence of a reinforcement learning process. TL is the process of reusing learned information across tasks. Information is shared between a source and a target task. Previous work has required that the target task wait until the source task has nished learning before transferring information. The execution of the source task prior to the target task considerably extends the time required for the target task to complete. This paper proposes a novel approach allowing both source and target task to learn in parallel. This allows the transfer to be bi-directional, so processes can act as both source and target simultaneously. This, in consequence, allows tasks to learn from each other’s experiences and thereby reduces the learning time required. The proposed approach is evaluated on a multi-agent smartgrid scenario.

[1]  Michail G. Lagoudakis,et al.  Coordinated Reinforcement Learning , 2002, ICML.

[2]  Siobhán Clarke,et al.  Reducing electricity costs in a dynamic pricing environment , 2012, 2012 IEEE Third International Conference on Smart Grid Communications (SmartGridComm).

[3]  Ioannis P. Vlahavas,et al.  Transfer Learning in Multi-Agent Reinforcement Learning Domains , 2011, EWRL.

[4]  B.F. Wollenberg,et al.  Toward a smart grid: power delivery for the 21st century , 2005, IEEE Power and Energy Magazine.

[5]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[6]  Haitham Bou-Ammar,et al.  Reinforcement Learning Transfer via Common Subspaces , 2011, ALA.

[7]  Warrren B Powell,et al.  A review of stochastic algorithms with continuous value function approximation and some new approximate policy iteration algorithms for multidimensional continuous applications , 2011 .

[8]  Manuela M. Veloso,et al.  Multiagent Systems: A Survey from a Machine Learning Perspective , 2000, Auton. Robots.

[9]  N. D. Hatziargyriou,et al.  Multi-agent reinforcement learning for microgrids , 2010, IEEE PES General Meeting.

[10]  Peter Stone,et al.  Transfer Learning for Reinforcement Learning Domains: A Survey , 2009, J. Mach. Learn. Res..

[11]  Reinaldo A. C. Bianchi,et al.  Combining independent and joint learning: a negotiation based approach , 2012, AAMAS.

[12]  Haitham Bou-Ammar,et al.  Reinforcement learning transfer via sparse coding , 2012, AAMAS.

[13]  K. Schneider,et al.  GridLAB-D: An open-source power systems modeling and simulation environment , 2008, 2008 IEEE/PES Transmission and Distribution Conference and Exposition.

[14]  Nikos Vlassis,et al.  A Concise Introduction to Multiagent Systems and Distributed AI , 2003 .

[15]  Siobhán Clarke,et al.  Management and control of energy usage and price using participatory sensing data , 2012 .

[16]  Peter Stone,et al.  Transfer Learning via Inter-Task Mappings for Temporal Difference Learning , 2007, J. Mach. Learn. Res..

[17]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[18]  C. V. Ramamoorthy,et al.  Knowledge and Data Engineering , 1989, IEEE Trans. Knowl. Data Eng..

[19]  Vinny Cahill,et al.  Autonomic multi-policy optimization in pervasive systems: Overview and evaluation , 2012, TAAS.

[20]  Bart De Schutter,et al.  A Comprehensive Survey of Multiagent Reinforcement Learning , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[21]  Chris Drummond,et al.  Accelerating Reinforcement Learning by Composing Solutions of Automatically Identified Subtasks , 2011, J. Artif. Intell. Res..

[22]  Gerhard Weiss,et al.  Multiagent Systems , 1999 .