Accelerating Nash Q-Learning with Graphical Game Representation and Equilibrium Solving

Traditional Nash Q-learning algorithm generally accepts a fact that agents are tightly coupled, which brings huge computing burden. However, many multi-agent systems in the real world have sparse interactions between agents. In this paper, sparse interactions are divided into two categories: intra-group sparse interactions and inter-group sparse interactions. Previous methods can only deal with one specific type of sparse interactions. Aiming at characterizing the two categories of sparse interactions, we use a novel mathematical model called Markov graphical game. On this basis, graphical game-based Nash Q-learning is proposed to deal with different types of interactions. Experimental results show that our algorithm takes less time per episode and acquires a good policy.

[1]  Michail G. Lagoudakis,et al.  Coordinated Reinforcement Learning , 2002, ICML.

[2]  Ming Zhou,et al.  Mean Field Multi-Agent Reinforcement Learning , 2018, ICML.

[3]  M. Bacharach.  Two-person Cooperative Games , 1976 .

[4]  Maja J. Mataric,et al.  Reward Functions for Accelerated Learning , 1994, ICML.

[5]  Zongqing Lu,et al.  Learning Attentional Communication for Multi-Agent Cooperation , 2018, NeurIPS.

[6]  Paul Scerri,et al.  Coordination of Large-Scale Multiagent Systems , 2005 .

[7]  Yang Gao,et al.  Multiagent Reinforcement Learning With Unshared Value Functions , 2015, IEEE Transactions on Cybernetics.

[8]  Keith B. Hall,et al.  Correlated Q-Learning , 2003, ICML.

[9]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[10]  Peter Vrancx,et al.  Learning multi-agent state space representations , 2010, AAMAS.

[11]  Manuela M. Veloso,et al.  Decentralized MDPs with sparse interactions , 2011, Artif. Intell..

[12]  Bart De Schutter,et al.  A Comprehensive Survey of Multiagent Reinforcement Learning , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[13]  Luis E. Ortiz,et al.  Nash Propagation for Loopy Graphical Games , 2002, NIPS.

[14]  J. Nash.  Equilibrium Points in N-Person Games. , 1950, Proceedings of the National Academy of Sciences of the United States of America.

[15]  Yujing Hu,et al.  Learning in Multi-agent Systems with Sparse Interactions by Knowledge Transfer and Game Abstraction , 2015, AAMAS.

[16]  Haris Aziz,et al.  Multiagent systems: algorithmic, game-theoretic, and logical foundations by Y. Shoham and K. Leyton-Brown Cambridge University Press, 2008 , 2010, SIGA.

[17]  Michael P. Wellman,et al.  Nash Q-Learning for General-Sum Stochastic Games , 2003, J. Mach. Learn. Res..

[18]  Michael L. Littman,et al.  Graphical Models for Game Theory , 2001, UAI.

[19]  Yang Gao,et al.  Multiagent Reinforcement Learning With Sparse Interactions by Negotiation and Knowledge Transfer , 2015, IEEE Transactions on Cybernetics.

[20]  Yoav Shoham,et al.  Multiagent Systems - Algorithmic, Game-Theoretic, and Logical Foundations , 2009 .

[21]  Yi Wu,et al.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments , 2017, NIPS.

[22]  Peter Vrancx,et al.  Solving Sparse Delayed Coordination Problems in Multi-Agent Reinforcement Learning , 2011, ALA.

[23]  Manuela M. Veloso,et al.  Learning of coordination: exploiting sparse interactions in multiagent systems , 2009, AAMAS.

[24]  Maja J. Mataric,et al.  Reinforcement Learning in the Multi-Robot Domain , 1997, Auton. Robots.

[25]  Andrew G. Barto,et al.  Improving Elevator Performance Using Reinforcement Learning , 1995, NIPS.

[26]  Georg Gottlob,et al.  Pure Nash equilibria: hard and easy games , 2003, TARK '03.

[27]  Fei Sha,et al.  Actor-Attention-Critic for Multi-Agent Reinforcement Learning , 2018, ICML.

[28]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.