Lateral Transfer Learning for Multiagent Reinforcement Learning

Some researchers have introduced transfer learning mechanisms to multiagent reinforcement learning (MARL). However, the existing works devoted to cross-task transfer for multiagent systems were designed just for homogeneous agents or similar domains. This work proposes an all-purpose cross-transfer method, called multiagent lateral transfer (MALT), assisting MARL with alleviating the training burden. We discuss several challenges in developing an all-purpose multiagent cross-task transfer learning method and provide a feasible way of reusing knowledge for MARL. In the developed method, we take features as the transfer object rather than policies or experiences, inspired by the progressive network. To achieve more efficient transfer, we assign pretrained policy networks for agents based on clustering, while an attention module is introduced to enhance the transfer framework. The proposed method has no strict requirements for the source task and target task. Compared with the existing works, our method can transfer knowledge among heterogeneous agents and also avoid negative transfer in the case of fully different tasks. As far as we know, this article is the first work denoted to all-purpose cross-task transfer for MARL. Several experiments in various scenarios have been conducted to compare the performance of the proposed method with baselines. The results demonstrate that the method is sufficiently flexible for most settings, including cooperative, competitive, homogeneous, and heterogeneous configurations.

[1]  Heechan Yang,et al.  Guided Soft Attention Network for Classification of Breast Cancer Histopathology Images , 2020, IEEE Transactions on Medical Imaging.

[2]  Zihan Zhou,et al.  Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning , 2020, ICLR.

[3]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[4]  Sumit Kumar,et al.  Learning Transferable Cooperative Behavior in Multi-Agent Teams , 2019, AAMAS.

[5]  Mark O. Riedl,et al.  Transfer in Deep Reinforcement Learning Using Knowledge Graphs , 2019, EMNLP.

[6]  Lin Zhao,et al.  Barrier Lyapunov functions-based command filtered output feedback control for full-state constrained nonlinear systems , 2019, Autom..

[7]  Zheng Li,et al.  A Multi-Agent Deep Reinforcement Learning Based Spectrum Allocation Framework for D2D Communications , 2019, 2019 IEEE Global Communications Conference (GLOBECOM).

[8]  Gang Liu,et al.  Bidirectional LSTM with attention mechanism and convolutional layer for text classification , 2019, Neurocomputing.

[9]  Felipe Leno da Silva,et al.  A Survey on Transfer Learning for Multiagent Reinforcement Learning Systems , 2019, J. Artif. Intell. Res..

[10]  Fei Sha,et al.  Actor-Attention-Critic for Multi-Agent Reinforcement Learning , 2018, ICML.

[11]  Yoav Goldberg,et al.  Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation , 2018, ICML.

[12]  Dongbin Zhao,et al.  StarCraft Micromanagement With Reinforcement Learning and Curriculum Transfer Learning , 2018, IEEE Transactions on Emerging Topics in Computational Intelligence.

[13]  Dirk Neumann,et al.  Transfer Learning versus Multiagent Learning regarding Distributed Decision-Making in Highway Traffic , 2018, ATT@IJCAI.

[14]  Yifeng Zhu,et al.  Zero Shot Transfer Learning for Robot Soccer , 2018, AAMAS.

[15]  Yee Whye Teh,et al.  Distral: Robust multitask reinforcement learning , 2017, NIPS.

[16]  Yi Wu,et al.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments , 2017, NIPS.

[17]  Sergey Levine,et al.  Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning , 2017, ICLR.

[18]  Razvan Pascanu,et al.  Sim-to-Real Robot Learning from Pixels with Progressive Nets , 2016, CoRL.

[19]  Yang Gao,et al.  Multiagent Reinforcement Learning With Sparse Interactions by Negotiation and Knowledge Transfer , 2015, IEEE Transactions on Cybernetics.

[20]  Risto Miikkulainen,et al.  Object-Model Transfer in the General Video Game Domain , 2016, AIIDE.

[21]  Eric Eaton,et al.  Using Task Features for Zero-Shot Knowledge Transfer in Lifelong Learning , 2016, IJCAI.

[22]  Razvan Pascanu,et al.  Progressive Neural Networks , 2016, ArXiv.

[23]  Qi Ye,et al.  Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation , 2016, ECCV.

[24]  Geoff S. Nitschke,et al.  Multi-agent Behavior-Based Policy Transfer , 2016, EvoApplications.

[25]  Francesco Buccafurri,et al.  Experimenting with Certified Reputation in a Competitive Multi-Agent Scenario , 2016, IEEE Intelligent Systems.

[26]  Qian Ma,et al.  Output consensus for heterogeneous multi-agent systems with linear dynamics , 2015, Appl. Math. Comput..

[27]  Malcolm I. Heywood,et al.  Knowledge Transfer from Keepaway Soccer to Half-field Offense through Program Symbiosis: Building Simple Programs for a Complex Task , 2015, GECCO.

[28]  Yoshua Bengio,et al.  Attention-Based Models for Speech Recognition , 2015, NIPS.

[29]  Peter Stone,et al.  Learning Inter-Task Transferability in the Absence of Target Task Samples , 2015, AAMAS.

[30]  Yang Gao,et al.  Multiagent Reinforcement Learning With Unshared Value Functions , 2015, IEEE Transactions on Cybernetics.

[31]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[32]  Lihong Li,et al.  PAC-inspired Option Discovery in Lifelong Reinforcement Learning , 2014, ICML.

[33]  Akiya Kamimura,et al.  Transfer Learning Method Using Ontology for Heterogeneous Multi-agent Reinforcement Learning , 2014 .

[34]  Jing Gao,et al.  On handling negative transfer and imbalanced distributions in multiple source transfer learning , 2014, SDM.

[35]  Tim Clarke,et al.  Transfer Learning: A Paradigm for Dynamic Spectrum and Topology Management in Flexible Architectures , 2013, 2013 IEEE 78th Vehicular Technology Conference (VTC Fall).

[36]  Siobhán Clarke,et al.  Transfer learning in multi-agent systems through parallel transfer , 2013 .

[37]  Subramanian Ramamoorthy,et al.  Comparative evaluation of MAL algorithms in a diverse set of ad hoc team problems , 2012, AAMAS.

[38]  Ioannis P. Vlahavas,et al.  Transfer Learning in Multi-Agent Reinforcement Learning Domains , 2011, EWRL.

[39]  Peter Vrancx,et al.  Transfer Learning for Multi-agent Coordination , 2011, ICAART.

[40]  Sarit Kraus,et al.  Ad Hoc Autonomous Agent Teams: Collaboration without Pre-Coordination , 2010, AAAI.

[41]  Gavriel Salomon,et al.  T RANSFER OF LEARNING , 1992 .