Anticipatory Behavior of Software Agents in Self-organizing Negotiations

Software agents are a well-established approach for modeling autonomous entities in distributed artificial intelligence. Iterated negotiations allow for coordinating the activities of multiple autonomous agents by means of repeated interactions. However, if several agents interact concurrently, the participants’ activities can mutually influence each other. This leads to poor coordination results. In this paper, we discuss these interrelations and propose a self-organization approach to cope with that problem. To that end, we apply distributed reinforcement learning as a feedback mechanism to the agents’ decision-making process. This enables the agents to use their experiences from previous activities to anticipate the results of potential future actions. They mutually adapt their behaviors to each other which results in the emergence of social order within the multiagent system. We empirically evaluate the dynamics of that process in a multiagent resource allocation scenario. The results show that the agents successfully anticipate the reactions to their activities in that dynamic and partially observable negotiation environment. This enables them to maximize their payoffs and to drastically outperform non-anticipating agents.

[1]  Jan D. Gehrke,et al.  Designing a Simulation Middleware for FIPA Multiagent Systems , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[2]  Jan Ole Berndt.  Self-Organizing Logistics Process Control: An Agent-Based Approach , 2011, ICAART.

[3]  N. R. Jennings,et al.  To appear in: Int Journal of Group Decision and Negotiation GDN2000 Keynote Paper Automated Negotiation: Prospects, Methods and Challenges , 2022 .

[4]  Jan Ole Berndt.  Self-organizing Supply Networks - Autonomous Agent Coordination based on Expectations , 2011, ICAART.

[5]  N. Luhmann.  Soziale Systeme : Grundriß einer allgemeinen Theorie , 1984 .

[6]  Ulrich Endriss,et al.  Nash Social Welfare in Multiagent Resource Allocation , 2009, AMEC/TADA.

[7]  Bart De Schutter,et al.  Multi-agent Reinforcement Learning: An Overview , 2010 .

[8]  Craig Boutilier,et al.  The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems , 1998, AAAI/IAAI.

[9]  J. Neumann,et al.  Theory of games and economic behavior , 1945, 100 Years of Math Milestones.

[10]  Yoav Shoham,et al.  Combinatorial Auctions , 2005, Encyclopedia of Wireless Networks.

[11]  Otthein Herzog,et al.  Efficient Multiagent Coordination in Dynamic Environments , 2011, 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology.

[12]  H. Maibom.  Social Systems , 2007 .

[13]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[14]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[15]  Otthein Herzog,et al.  Distributed Learning of Best Response Behaviors in Concurrent Iterated Many-Object Negotiations , 2012, MATES.

[16]  N. Luhmann.  Probleme mit operativer Schließung , 2005 .

[17]  Johannes A. La Poutre,et al.  Why Agents for Automated Negotiations Should Be Adaptive , 2003 .

[18]  Yoav Shoham,et al.  Simple search methods for finding a Nash equilibrium , 2004, Games Econ. Behav..

[19]  G. Oppenheim,et al.  A Guided Tour , 2007 .

[20]  Sven Werner,et al.  Distributed Clustering of Autonomous Shipping Containers by Concept, Location, and Time , 2007, MATES.

[21]  J. Neumann.  Zur Theorie der Gesellschaftsspiele , 1928 .

[22]  Nicolas Maudet,et al.  Negotiating Socially Optimal Allocations of Resources , 2011, J. Artif. Intell. Res..

[23]  Nicholas R. Jennings,et al.  Negotiation decision functions for autonomous agents , 1998, Robotics Auton. Syst..

[24]  Nicholas R. Jennings,et al.  Intelligent agents: theory and practice , 1995, The Knowledge Engineering Review.

[25]  Otthein Herzog,et al.  Distributed Reinforcement Learning for Optimizing Resource Allocation in Autonomous Logistics Processes , 2012, LDIC.

[26]  Arne Schuldt,et al.  Multiagent Coordination Enabling Autonomous Logistics , 2011, KI - Künstliche Intelligenz.

[27]  John Dickhaut,et al.  Price Formation in Double Auctions , 2001, E-Commerce Agents.

[28]  Nicholas R. Jennings,et al.  The Cooperative Problem-solving Process , 1999, J. Log. Comput..

[29]  Otthein Herzog,et al.  The Interaction Effort in Autonomous Logistics Processes: Potential and Limitations for Cooperation , 2011 .

[30]  J. Nash.  NON-COOPERATIVE GAMES , 1951, Classics in Game Theory.

[31]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[32]  Keki B. Irani,et al.  An Algorithmic Solution of N-Person Games , 1986, AAAI.