Adaptivity in agent-based routing for data networks

Adaptivity, both of the individual agents and of the interaction structure among the agents, seems indispensable for scaling up multi-agent systems (MAS s) in noisy environments. One important consideration in designing adaptive agents is choosing their action spaces to be as amenable as possible to machine learning techniques, especially to reinforcement learning (RL) techniques. One important way to have the interaction structure connecting agents itself be adaptive is to have the intentions and/or actions of the agents be in the input spaces of the other agents, much as in Stackelberg games. We consider both kinds of adaptivity in the design of a MAS to control network packet routing. We demonstrate on the OPNET event-driven network simulator the perhaps surprising fact that simply changing the action space of the agents to be better suited to RL can result in very large improvements in their potential performance: at their best settings, our learning-amenable router agents achieve throughputs up to three and one half times better than that of the standard Bellman-Ford routing algorithm, even when the Bellman-Ford protocol traffic is maintained. We then demonstrate that much of that potential improvement can be realized by having the agents learn their settings when the agent interaction structure is itself adaptive.

[1]  Narsingh Deo,et al.  Shortest-path algorithms: Taxonomy and annotation , 1984, Networks.

[2]  Dimitri P. Bertsekas,et al.  Data Networks , 1986 .

[3]  Kagan Tumer,et al.  An Introduction to Collective Intelligence , 1999, ArXiv.

[4]  Edsger W. Dijkstra,et al.  A note on two problems in connexion with graphs , 1959, Numerische Mathematik.

[5]  Martin Heusse,et al.  Adaptive Agent-Driven Routing and Load Balancing in Communication Networks , 1998, Adv. Complex Syst..

[6]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[7]  Craig Boutilier,et al.  The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems , 1998, AAAI/IAAI.

[8]  A. Roadmapof.  A Roadmap of Agent Research and Development , 1995 .

[9]  Ariel Orda,et al.  Virtual path bandwidth allocation in multiuser networks , 1997, TNET.

[10]  Kagan Tumer,et al.  Avoiding Braess' Paradox through Collective Intelligence , 1999, ArXiv.

[11]  Ariel Orda,et al.  Capacity allocation under noncooperative routing , 1997, IEEE Trans. Autom. Control..

[12]  Ariel Orda,et al.  Achieving network optima using Stackelberg routing strategies , 1997, TNET.

[13]  Kagan Tumer,et al.  Using Collective Intelligence to Route Internet Traffic , 1998, NIPS.

[14]  Michael P. Wellman,et al.  Multiagent Reinforcement Learning: Theoretical Framework and an Algorithm , 1998, ICML.

[15]  Claudia V. Goldman,et al.  Emergent Coordination through the Use of Cooperative State-Changing Rules , 1994, AAAI.

[16]  Ariel Orda,et al.  Architecting noncooperative networks , 1995, Eighteenth Convention of Electrical and Electronics Engineers in Israel.

[17]  Michael L. Littman,et al.  Packet Routing in Dynamically Changing Networks: A Reinforcement Learning Approach , 1993, NIPS.

[18]  K. Sycara,et al.  This Is a Publication of the American Association for Artificial Intelligence Multiagent Systems Multiagent System Issues and Challenges Individual Agent Reasoning Task Allocation Multiagent Planning Recognizing and Resolving Conflicts Managing Communication Modeling Other Agents Managing Resources , 2022 .

[19]  Devika Subramanian,et al.  Ants and Reinforcement Learning: A Case Study in Routing in Dynamic Networks , 1997, IJCAI.

[20]  Richard Bellman,et al.  ON A ROUTING PROBLEM , 1958 .

[21]  D. R. Fulkerson,et al.  Maximal Flow Through a Network , 1956 .

[22]  T. Başar,et al.  Dynamic Noncooperative Game Theory , 1982 .

[23]  Victor R. Lesser,et al.  Coalitions Among Computationally Bounded Agents , 1997, Artif. Intell..

[24]  Gerhard Weiss,et al.  Multiagent Systems , 1999 .

[25]  Kagan Tumer,et al.  General principles of learning-based multi-agent systems , 1999, AGENTS '99.

[26]  Shailesh Kumar and Risto Miikkulainen.  Dual Reinforcement Q-Routing: An On-Line Adaptive Routing Algorithm , 1997 .