Creating large numbers of game AIs by learning behavior for cooperating units

We present two improvements to the hybrid learning method for the shout-ahead architecture for units in the game Battle for Wesnoth. The shout-ahead architecture allows for units to perform decision making in two stages, first determining an action without knowledge of the intentions of other units, then, after communicating the intended action and likewise receiving the intentions of the other units, taking these intentions into account for the final decision on the next action. The decision making uses two rule sets and reinforcement learning is used to learn rule weights (that influence decision making), while evolutionary learning is used to evolve good rule sets. Our improvements add knowledge about terrain to the learning and also evaluate unit behaviors on several scenario maps to learn more general rules. The use of terrain knowledge resulted in improvements in the win percentage of evolved teams between 3 and 14 percentage points for different maps, while using several maps to learn from resulted in nearly similar win percentages on maps not learned from as on the maps learned from.

[1]  Marc J. V. Ponsen,et al.  Improving Adaptive Game Ai with Evolutionary Learning , 2004 .

[2]  Simon M. Lucas,et al.  Learning and evolving combat game controllers , 2012, 2012 IEEE Conference on Computational Intelligence and Games (CIG).

[3]  Jörg Denzinger,et al.  Behavior Learning-Based Testing of Starcraft Competition Entries , 2011, AIIDE.

[4]  Simon M. Lucas,et al.  Evolution versus Temporal Difference Learning for learning to play Ms. Pac-Man , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[5]  Nathan R. Sturtevant,et al.  Memory-Efficient Abstractions for Pathfinding , 2007, AIIDE.

[6]  Kenneth O. Stanley and Bobby D. Bryant and Risto Miikkulainen,et al.  Real-Time Evolution in the NERO Video Game (Winner of CIG 2005 Best Paper Award) , 2005, CIG.

[7]  Yan Li,et al.  Evolutionary neural network for ghost in Ms. Pac-Man , 2011, 2011 International Conference on Machine Learning and Cybernetics.

[8]  Regina Barzilay,et al.  Learning to Win by Reading Manuals in a Monte-Carlo Framework , 2011, ACL.

[9]  Julian Togelius,et al.  Search-Based Procedural Content Generation: A Taxonomy and Survey , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[10]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[11]  Gita Reese Sukthankar,et al.  Learning Policies for First Person Shooter Games Using Inverse Reinforcement Learning , 2011, AIIDE.

[12]  Jörg Denzinger,et al.  Improving testing of multi-unit computer players for unwanted behavior using coordination macros , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[13]  Risto Miikkulainen,et al.  Human-assisted neuroevolution through shaping, advice and examples , 2011, GECCO '11.

[14]  Ashwin Ram,et al.  Transfer Learning in Real-Time Strategy Games Using Hybrid CBR/RL , 2007, IJCAI.

[15]  Jörg Denzinger,et al.  A Hybrid Cooperative Behavior Learning Method for a Rule-Based Shout-Ahead Architecture , 2012, 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology.

[16]  Marcus Gallagher,et al.  Learning to play Pac-Man: an evolutionary, rule-based approach , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..