State-coupled replicator dynamics

This paper introduces a new model, i.e. state-coupled replicator dynamics, expanding the link between evolutionary game theory and multiagent reinforcement learning to multistate games. More precisely, it extends and improves previous work on piecewise replicator dynamics, a combination of replicators and piecewise models. The contributions of the paper are twofold. One, we identify and explain the major shortcomings of piecewise replicators, i.e. discontinuities and occurrences of qualitative anomalies. Two, this analysis leads to the proposal of the new model for learning dynamics in stochastic games, named state-coupled replicator dynamics. The preceding formalization of piecewise replicators - general in the number of agents and states - is factored into the new approach. Finally, we deliver a comparative study of finite action-set learning automata to piecewise and state-coupled replicator dynamics. Results show that state-coupled replicators model learning dynamics in stochastic games more accurately than their predecessor, the piecewise approach.

[1]  Kumpati S. Narendra,et al.  Learning automata - an introduction , 1989 .

[2]  Peter Vrancx,et al.  Networks of Learning Automata and Limiting Games , 2007, Adaptive Agents and Multi-Agents Systems.

[3]  Tilman Börgers,et al.  Learning Through Reinforcement and Replicator Dynamics , 1997 .

[4]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[5]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[6]  Matthias Rauterberg,et al.  Formalizing Multi-state Learning Dynamics , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[7]  Tom Lenaerts,et al.  A selection-mutation model for q-learning in multi-agent systems , 2003, AAMAS '03.

[8]  Peter Vrancx,et al.  Switching dynamics of multi-agent learning , 2008, AAMAS.

[9]  Simon Parsons,et al.  What evolutionary game theory tells us about multiagent learning , 2007, Artif. Intell..

[10]  Karl Tuyls,et al.  Theoretical Advantages of Lenient Learners: An Evolutionary Game Theoretic Perspective , 2008, J. Mach. Learn. Res..