To cope with large scale, agents are usually organized in a network such that an agent interacts only with its immediate neighbors in the network. Reinforcement learning techniques have been commonly used to optimize agents local policies in such a network because they require little domain knowledge and can be fully distributed. However, all of the previous work assumed the underlying network was fixed throughout the learning process. This assumption was important because the underlying network defines the learning context of each agent. In particular, the set of actions and the state space for each agent is defined in terms of the agent's neighbors. If agents dynamically change the underlying network structure (also called self-organize) during learning, then one needs a mechanism for transferring what agents have learned so far before (in the old network structure) to their new learning context (in the new network structure).
In this work we develop a novel self-organization mechanism that not only allows agents to self-organize the underlying network during the learning process, but also uses information from learning to guide the self-organization process. Consequently, our work is the first to study this interaction between learning and self-organization. Our self-organization mechanism uses heuristics to transfer the learned knowledge across the different steps of self-organization. We also present a more restricted version of our mechanism that is computationally less expensive and still achieve good performance. We use a simplified version of the distributed task allocation domain as our case study. Experimental results verify the stability of our approach and show a monotonic improvement in the performance of the learning process due to self-organization.
[1]
Claudia V. Goldman,et al.
Self-organization through bottom-up coalition formation
,
2003,
AAMAS '03.
[2]
Victor R. Lesser,et al.
Learning the task allocation game
,
2006,
AAMAS '06.
[3]
Michael H. Bowling,et al.
Convergence and No-Regret in Multiagent Learning
,
2004,
NIPS.
[4]
Marie desJardins,et al.
Agent-organized networks for dynamic team formation
,
2005,
AAMAS '05.
[5]
Manuela M. Veloso,et al.
Multiagent learning using a variable learning rate
,
2002,
Artif. Intell..
[6]
Michael L. Littman,et al.
Packet Routing in Dynamically Changing Networks: A Reinforcement Learning Approach
,
1993,
NIPS.
[7]
Peter Stone,et al.
Behavior transfer for value-function-based reinforcement learning
,
2005,
AAMAS '05.
[8]
Bryan Horling,et al.
Quantitative organizational modeling and design for multi-agent systems
,
2006
.
[9]
Leonid Peshkin,et al.
Reinforcement learning for adaptive routing
,
2002,
Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).