Switching dynamics of multi-agent learning

This paper presents the dynamics of multi-agent reinforcement learning in multiple state problems. We extend previous work that formally modelled the relation between reinforcement learning agents and replicator dynamics in stateless multi-agent games. More precisely, in this work we use a combination of replicator dynamics and switching dynamics to model multi-agent learning automata in multi-state games. This is the first time that the dynamics of problems with more than one state is considered with replicator equations. Previously, it was unclear how the replicator dynamics of stateless games had to be extended to account for multiple states. We use our model to visualize the basin of attraction of the learning agents and the boundaries of switching dynamics at which an agent possibly arrives in a new dynamical system. Our model allows to analyze and predict the behavior of the different learning agents in a wide variety of multi-state problems. In our experiments we illustrate this powerful method in two games with two agents and two states.

[1]  L. Glass,et al.  The logical analysis of continuous, non-linear biochemical control networks. , 1973, Journal of theoretical biology.

[2]  D. Sherrington.  Stochastic Processes in Physics and Chemistry , 1983 .

[3]  K. Narendra,et al.  Decentralized learning in finite Markov chains , 1985, 1985 24th IEEE Conference on Decision and Control.

[4]  Jörgen W. Weibull,et al.  Evolutionary Game Theory , 1996 .

[5]  Josef Hofbauer,et al.  Evolutionary Games and Population Dynamics , 1998 .

[6]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[7]  Tom Lenaerts,et al.  A selection-mutation model for q-learning in multi-agent systems , 2003, AAMAS '03.

[8]  M. Thathachar,et al.  Networks of Learning Automata: Techniques for Online Stochastic Optimization , 2003 .

[9]  H. D. Jong,et al.  Qualitative simulation of genetic regulatory networks using piecewise-linear models , 2004, Bulletin of mathematical biology.

[10]  Karl Tuyls,et al.  An Evolutionary Dynamical Analysis of Multi-Agent Learning in Iterated Games , 2005, Autonomous Agents and Multi-Agent Systems.

[11]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[12]  Yoav Shoham,et al.  If multi-agent learning is the answer, what is the question? , 2007, Artif. Intell..

[13]  Karl Tuyls,et al.  Theoretical advantages of lenient Q-learners: an evolutionary game theoretic perspective , 2007, AAMAS '07.

[14]  Peter Vrancx,et al.  Networks of Learning Automata and Limiting Games , 2007, Adaptive Agents and Multi-Agents Systems.

[15]  Simon Parsons,et al.  What evolutionary game theory tells us about multiagent learning , 2007, Artif. Intell..

[16]  C. Cannings,et al.  Evolutionary Game Theory , 2010 .