Learning when and how to coordinate

This paper examines the potential and the impact of introducing learning capabilities into autonomous agents that make decisions at run-time about which mechanism to exploit in order to coordinate their activities. Specifically, the efficacy of learning is evaluated for making the decisions that are involved in determining when and how to coordinate. Our motivating hypothesis is that to deal with dynamic and unpredictable environments it is important to have agents that can learn the right situations in which to attempt to coordinate and the right method to use in those situations. This hypothesis is evaluated empirically, using reinforcement based algorithms, in a grid-world scenario in which a) an agent's predictions about the other agents in the environment are approximately correct and b) an agent can not correctly predict the others' behaviour. The results presented show when, where and why learning is effective when it comes to making a decision about selecting a coordination mechanism.

[1]  Tommi S. Jaakkola,et al.  Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms , 2000, Machine Learning.

[2]  Victor R. Lesser,et al.  Learning to Improve Coordinated Actions in Cooperative Distributed Problem-Solving Environments , 1998, Machine Learning.

[3]  Victor R. Lesser,et al.  Learning Situation-Specific Coordination in Cooperative Multi-agent Systems , 1999, Autonomous Agents and Multi-Agent Systems.

[4]  Edmund H. Durfee.  Practically Coordinating , 1999, AI Mag..

[5]  Nicholas R. Jennings,et al.  The Dynamic Selection of Coordination Mechanisms , 2004, Autonomous Agents and Multi-Agent Systems.

[6]  Edmund H. Durfee,et al.  Partial global planning: a coordination framework for distributed hypothesis formation , 1991, IEEE Trans. Syst. Man Cybern..

[7]  Edmund H. Durfee,et al.  Agents Learning about Agents: A Framework and Analysis , 1997 .

[8]  Michael P. Wellman,et al.  Online learning about other agents in a dynamic multiagent system , 1998, AGENTS '98.

[9]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[10]  Jay R. Galbraith.  Designing Complex Organizations , 1973 .

[11]  Craig Boutilier,et al.  Sequential Optimality and Coordination in Multiagent Systems , 1999, IJCAI.

[12]  Manuela M. Veloso,et al.  Multiagent Systems: A Survey from a Machine Learning Perspective , 2000, Auton. Robots.

[13]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[14]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[15]  Nicholas R. Jennings,et al.  Learning to select a coordination mechanism , 2002, AAMAS '02.

[16]  K. Suzanne Barber,et al.  Coordinating Distributed Decision Making Using Reusable Interaction Specifications , 2000, PRIMA.

[17]  Mark S. Fox,et al.  An Organizational View of Distributed Systems , 1988, IEEE Transactions on Systems, Man, and Cybernetics.

[18]  Nicholas R. Jennings,et al.  Run-Time Selection of Coordination Mechanisms in Multi-Agent Systems , 2000, ECAI.

[19]  Shin Ishii,et al.  Multi-agent reinforcement learning: an approach based on the other agent's internal model , 2000, Proceedings Fourth International Conference on MultiAgent Systems.

[20]  Leonid Sheremetov,et al.  Weiss, Gerhard. Multiagent Systems a Modern Approach to Distributed Artificial Intelligence , 2009 .

[21]  Sandip Sen,et al.  Learning to Coordinate without Sharing Information , 1994, AAAI.

[22]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[23]  Craig Boutilier,et al.  The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems , 1998, AAAI/IAAI.

[24]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[25]  Victor R. Lesser,et al.  Reflections on the Nature of Multi-Agent Coordination and Its Implications for an Agent Architecture , 2004, Autonomous Agents and Multi-Agent Systems.

[26]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[27]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[28]  Nicholas R. Jennings,et al.  Commitments and conventions: The foundation of coordination in multi-agent systems , 1993, The Knowledge Engineering Review.

[29]  Randall Davis,et al.  Frameworks for Cooperation in Distributed Problem Solving , 1988, IEEE Transactions on Systems, Man, and Cybernetics.

[30]  Moshe Tennenholtz,et al.  On the Synthesis of Useful Social Laws for Artificial Agent Societies (Preliminary Report) , 1992, AAAI.

[31]  Nicholas R. Jennings,et al.  Reasoning about commitments and penalties for coordination between autonomous agents , 2001, AGENTS '01.

[32]  Michael Wooldridge,et al.  Agent-based software engineering , 1997, IEE Proc. Softw. Eng..

[33]  Thomas W. Malone,et al.  Modeling Coordination in Organizations and Markets , 1987 .

[34]  Sandip Sen,et al.  Learning in multiagent systems , 1999 .

[35]  S.J.J. Smith,et al.  Empirical Methods for Artificial Intelligence , 1995 .

[36]  Jeffrey S. Rosenschein,et al.  Rules of Encounter - Designing Conventions for Automated Negotiation among Computers , 1994 .