Multi-agent Relational Reinforcement Learning

In this paper we report on using a relational state space in multi-agent reinforcement learning. There is growing evidence in the Reinforcement Learning research community that a relational representation of the state space has many benefits over a propositional one. Complex tasks as planning or information retrieval on the web can be represented more naturally in relational form. Yet, this relational structure has not been exploited for multi-agent reinforcement learning tasks and has only been studied in a single agent context so far. In this paper we explore the powerful possibilities of using Relational Reinforcement Learning (RRL) in complex multi-agent coordination tasks. More precisely, we consider an abstract multi-state coordination problem, which can be considered as a variation and extension of repeated stateless Dispersion Games. Our approach shows that RRL allows to represent a complex state space in a multi-agent environment more compactly and allows for fast convergence of learning agents. Moreover, with this technique, agents are able to make complex interactive models (in the sense of learning from an expert), to predict what other agents will do and generalize over this model. This enables to solve complex multi-agent planning tasks, in which agents need to be adaptive and learn, with more powerful tools.

[1]  Maurice Bruynooghe,et al.  Towards Informed Reinforcement Learning , 2004, ICML 2004.

[2]  Mehdi Dastani,et al.  A characterization of sapient agents , 2003, IEMC '03 Proceedings. Managing Technologically Driven Organizations: The Human Side of Innovation and Change (IEEE Cat. No.03CH37502).

[3]  Craig Boutilier,et al.  The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems , 1998, AAAI/IAAI.

[4]  De,et al.  Relational Reinforcement Learning , 2022 .

[5]  Robert Givan,et al.  Relational Reinforcement Learning: An Overview , 2004, ICML 2004.

[6]  Saso Dzeroski,et al.  Integrating Experimentation and Guidance in Relational Reinforcement Learning , 2002, ICML.

[7]  Saso Dzeroski,et al.  Integrating Guidance into Relational Reinforcement Learning , 2004, Machine Learning.

[8]  Edmund H. Durfee,et al.  Agents Learning about Agents: A Framework and Analysis , 1997 .

[9]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[10]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[11]  Jan Ramon.  On the convergence of reinforcement learning using a decision tree learner , 2005, ICML 2005.

[12]  Peter Stone,et al.  Layered Learning in Multiagent Systems , 1997, AAAI/IAAI.

[13]  Tom Lenaerts,et al.  A selection-mutation model for q-learning in multi-agent systems , 2003, AAMAS '03.

[14]  Ann Nowé,et al.  Social Agents Playing a Periodical Policy , 2001, ECML.

[15]  Michael P. Wellman,et al.  Experimental Results on Q-Learning for General-Sum Stochastic Games , 2000, ICML.

[16]  Kagan Tumer,et al.  Collective Intelligence and Braess' Paradox , 2000, AAAI/IAAI.

[17]  K. Tuyls,et al.  Multi-Agent Relational Reinforcement Learning Explorations in Multi-State Coordination Tasks , 2006 .

[18]  Yoav Shoham,et al.  Dispersion games: general definitions and some specific learning results , 2002, AAAI/IAAI.

[19]  Luc De Raedt,et al.  Machine Learning: ECML 2001 , 2001, Lecture Notes in Computer Science.

[20]  Kurt Driessens,et al.  Speeding Up Relational Reinforcement Learning through the Use of an Incremental First Order Decision Tree Learner , 2001, ECML.

[21]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[22]  Richard S. Sutton,et al.  Reinforcement Learning , 1992, Handbook of Machine Learning.

[23]  Luc De Raedt,et al.  Logical Markov Decision Programs , 2003 .

[24]  Peter Stone,et al.  Layered learning in multiagent systems - a winning approach to robotic soccer , 2000, Intelligent robotics and autonomous agents.

[25]  Jan Ramon,et al.  Opponent modeling by analysing play , 2002 .

[26]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[27]  Pavel Brazdil,et al.  Proceedings of the European Conference on Machine Learning , 1993 .

[28]  Karl Tuyls,et al.  Analyzing Multi-agent Reinforcement Learning Using Evolutionary Dynamics , 2004, ECML.

[29]  Eduardo F. Morales,et al.  Learning to fly by combining reinforcement learning with behavioural cloning , 2004, ICML.

[30]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[31]  Sandip Sen,et al.  Towards a pareto-optimal solution in general-sum games , 2003, AAMAS '03.

[32]  Luc De Raedt,et al.  Inductive Verification and Validation of the KULRoT RoboCup Team , 1998, RoboCup.