Evolutionary game theory and multi-agent reinforcement learning

In this paper we survey the basics of reinforcement learning and (evolutionary) game theory, applied to the field of multi-agent systems. This paper contains three parts. We start with an overview on the fundamentals of reinforcement learning. Next we summarize the most important aspects of evolutionary game theory. Finally, we discuss the state-of-the-art of multi-agent reinforcement learning and the mathematical connection with evolutionary game theory.

[1]  Jörgen W. Weibull,et al.  Evolutionary Game Theory , 1996 .

[2]  Daniel Kudenko,et al.  Reinforcement learning of coordination in cooperative multi-agent systems , 2002, AAAI/IAAI.

[3]  Leslie Pack Kaelbling,et al.  Acting Optimally in Partially Observable Stochastic Domains , 1994, AAAI.

[4]  Tom Lenaerts,et al.  Learning to Reach the Pareto Optimal Nash Equilibrium as a Team , 2002, Australian Joint Conference on Artificial Intelligence.

[5]  Kagan Tumer,et al.  General principles of learning-based multi-agent systems , 1999, AGENTS '99.

[6]  M. L. Tsetlin,et al.  Automaton theory and modeling of biological systems , 1973 .

[7]  J. M. Smith,et al.  The Logic of Animal Conflict , 1973, Nature.

[8]  Craig Boutilier,et al.  The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems , 1998, AAAI/IAAI.

[9]  Frederick Mosteller,et al.  Stochastic Models for Learning , 1956 .

[10]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[11]  Gunes Ercal,et al.  On No-Regret Learning, Fictitious Play, and Nash Equilibrium , 2001, ICML.

[12]  Gerhard Weiß,et al.  Distributed reinforcement learning , 1995, Robotics Auton. Syst..

[13]  Peter Stone,et al.  Layered Learning in Multiagent Systems , 1997, AAAI/IAAI.

[14]  J. Weibull,et al.  Nash Equilibrium and Evolution by Imitation , 1994 .

[15]  J. Neumann,et al.  Theory of Games and Economic Behavior. , 1945 .

[16]  T. D. Schneider,et al.  Evolution of biological information. , 2000, Nucleic acids research.

[17]  P. S. Sastry,et al.  Varieties of learning automata: an overview , 2002, IEEE Trans. Syst. Man Cybern. Part B.

[18]  Michael P. Wellman,et al.  Multiagent Reinforcement Learning in Stochastic Games , 1999, ICML 1999.

[19]  Mark Perlman,et al.  The Rational Foundations of Economic Behaviour , 1996 .

[20]  Tilman Börgers,et al.  Learning Through Reinforcement and Replicator Dynamics , 1997 .

[21]  C.C. White,et al.  Dynamic programming and stochastic control , 1978, Proceedings of the IEEE.

[22]  Tom Lenaerts,et al.  A selection-mutation model for q-learning in multi-agent systems , 2003, AAMAS '03.

[23]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[24]  Sandip Sen,et al.  Towards a pareto-optimal solution in general-sum games , 2003, AAMAS '03.

[25]  Barbara Messing,et al.  An Introduction to MultiAgent Systems , 2002, Künstliche Intell..

[26]  Fernando Redondo.  Game Theory and Economics , 2001 .

[27]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Stochastic Control , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[28]  D. Stauffer.  Life, Love and Death: Models of Biological Reproduction and Aging , 1999 .

[29]  Gerhard Weiss,et al.  Multiagent Systems , 1999 .

[30]  D. Serra,et al.  Game theory and economics , 2003 .

[31]  Yoav Shoham,et al.  Dispersion games: general definitions and some specific learning results , 2002, AAAI/IAAI.

[32]  M. L. Tsetlin.  On the Behavior of Finite Automata in Random Media , 1961 .

[33]  W. Ames.  Mathematics in Science and Engineering , 1999 .

[34]  Ville Könönen.  Multiagent reinforcement learning in Markov games : asymmetric and symmetric approaches , 2004 .

[35]  J M Smith,et al.  Evolution and the theory of games , 1976 .

[36]  Theodore J. Perkins,et al.  On the Existence of Fixed Points for Q-Learning and Sarsa in Partially Observable Domains , 2002, ICML.

[37]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[38]  Ariel Rubinstein,et al.  A Course in Game Theory , 1995 .

[39]  Kagan Tumer,et al.  Using Collective Intelligence to Route Internet Traffic , 1998, NIPS.

[40]  Julie A. Adams,et al.  Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence , 2001, AI Mag..

[41]  Kumpati S. Narendra,et al.  Learning Automata - A Survey , 1974, IEEE Trans. Syst. Man Cybern..

[42]  Tom Lenaerts,et al.  Towards a relation between learning agents and evolutionary dynamics , 2002 .

[43]  Josef Hofbauer,et al.  Evolutionary Games and Population Dynamics , 1998 .

[44]  L. Samuelson.  Evolutionary Games and Equilibrium Selection , 1997 .

[45]  Charles W. Anderson,et al.  Strategy Learning with Multilayer Connectionist Representations , 1987 .

[46]  Mark D. Pendrith,et al.  An Analysis of Direct Reinforcement Learning in Non-Markovian Domains , 1998, ICML.

[47]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[48]  Kumpati S. Narendra,et al.  Learning automata - an introduction , 1989 .

[49]  John N. Tsitsiklis,et al.  Asynchronous stochastic approximation and Q-learning , 1993, Proceedings of 32nd IEEE Conference on Decision and Control.

[50]  Bernard Manderick,et al.  Extended Replicator Dynamics as a Key to Reinforcement Learning in Multi-agent Systems , 2003, ECML.

[51]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[52]  Martin Lauer,et al.  An Algorithm for Distributed Reinforcement Learning in Cooperative Multi-Agent Systems , 2000, ICML.

[53]  J. Neumann,et al.  The Theory of Games and Economic Behaviour , 1944 .

[54]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[55]  Karl Tuyls,et al.  On a Dynamical Analysis of Reinforcement Learning in Games: Emergence of Occam's Razor , 2003, CEEMAS.

[56]  Ann Nowé,et al.  Social Agents Playing a Periodical Policy , 2001, ECML.

[57]  M. Hirsch,et al.  Differential Equations, Dynamical Systems, and Linear Algebra , 1974 .