Learning cooperative behavior for the shout-ahead architecture

We present an agent architecture and a hybrid behavior learning method for it that allows the use of communicated intentions of other agents to create agents that are able to cooperate with various configurations of other agents in fulfilling a task. Our shout-ahead architecture is based on two rule sets, one making decisions without communicated intentions and one with these intentions, and reinforcement learning is used to determine in a particular situation which set is responsible for the final decision. Evolutionary learning is used to learn these rules. Our application of this approach to learning behaviors for units in a computer game shows that the use of shout-ahead using only communicated intentions in the second rule set substantially improves the quality of the learned behavior compared to agents not using shout-ahead. Also, allowing for additional conditions in the second rule set can either improve the quality or worsen it, based on what type of conditions are used.

[1]  Risto Miikkulainen,et al.  COOPERATIVE COEVOLUTION OF MULTI-AGENT SYSTEMS , 2001 .

[2]  Matthias Fuchs,et al.  Experiments in learning prototypical situations for variants of the pursuit game , 1999 .

[3]  Risto Miikkulainen,et al.  Human-assisted neuroevolution through shaping, advice and examples , 2011, GECCO '11.

[4]  Jörg Denzinger,et al.  Improving Evolutionary Learning of Cooperative Behavior by Including Accountability of Strategy Components , 2003, MATES.

[5]  Gita Reese Sukthankar,et al.  Learning Policies for First Person Shooter Games Using Inverse Reinforcement Learning , 2011, AIIDE.

[6]  Simon M. Lucas,et al.  Learning and evolving combat game controllers , 2012, 2012 IEEE Conference on Computational Intelligence and Games (CIG).

[7]  Yan Li,et al.  Evolutionary neural network for ghost in Ms. Pac-Man , 2011, 2011 International Conference on Machine Learning and Cybernetics.

[8]  Regina Barzilay,et al.  Learning to Win by Reading Manuals in a Monte-Carlo Framework , 2011, ACL.

[9]  Kenneth O. Stanley and Bobby D. Bryant and Risto Miikkulainen,et al.  Real-Time Evolution in the NERO Video Game (Winner of CIG 2005 Best Paper Award) , 2005, CIG.

[10]  Jörg Denzinger,et al.  On cooperation between evolutionary algorithms and other search paradigms , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[11]  Victor R. Lesser,et al.  Functionally Accurate, Cooperative Distributed Systems , 1988, IEEE Transactions on Systems, Man, and Cybernetics.

[12]  Claudia V. Goldman,et al.  Optimizing information exchange in cooperative multi-agent systems , 2003, AAMAS '03.

[13]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[14]  Sean Luke,et al.  Cooperative Multi-Agent Learning: The State of the Art , 2005, Autonomous Agents and Multi-Agent Systems.

[15]  Dipti Srinivasan,et al.  Cooperative, hybrid agent architecture for real-time traffic signal control , 2003, IEEE Trans. Syst. Man Cybern. Part A.

[16]  Alain Dutech,et al.  Cooperation through communication in decentralized Markov games , 2004 .

[17]  Ashwin Ram,et al.  Transfer Learning in Real-Time Strategy Games Using Hybrid CBR/RL , 2007, IJCAI.

[18]  Jörg Denzinger,et al.  A Hybrid Cooperative Behavior Learning Method for a Rule-Based Shout-Ahead Architecture , 2012, 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology.

[19]  Marcus Gallagher,et al.  Learning to play Pac-Man: an evolutionary, rule-based approach , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[20]  D. Vengerov,et al.  Learning, Cooperation, and Coordination in Multi-Agent Systems , 2000 .

[21]  Randall Davis,et al.  Negotiation as a Metaphor for Distributed Problem Solving , 1988, Artif. Intell..

[22]  Simon M. Lucas,et al.  Evolution versus Temporal Difference Learning for learning to play Ms. Pac-Man , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.