Evolutionary MCTS for Multi-Action Adversarial Games

Turn-based multi-action adversarial games are games in which each player turn consists of a sequence of atomic actions, resulting in an extremely high branching factor. Many strategy board, card, and video games fall into this category, for which the current state of the art is Online Evolutionary Planning (OEP) – an evolutionary algorithm (EA) that treats atomic actions as genes, and complete action sequences as genomes. In this paper, we introduce Evolutionary Monte Carlo Tree Search (EMCTS) to tackle this challenge, combining the tree search of MCTS with the sequence-based optimization of EAs. Experiments on the game Hero Academy show that EMCTS convincingly outperforms several baselines including OEP and an improved variant of OEP introduced in this paper, at different time settings and numbers of atomic actions per turn. EMCTS also scales better than any existing algorithm with the complexity of the problem.

[1]  Simon M. Lucas,et al.  Using genetic programming to evolve heuristics for a Monte Carlo Tree Search Ms Pac-Man agent , 2013, 2013 IEEE Conference on Computational Inteligence in Games (CIG).

[2]  Julian Togelius,et al.  Portfolio Online Evolution in StarCraft , 2016, AIIDE.

[3]  Simon M. Lucas,et al.  Knowledge-based fast evolutionary MCTS for general video game playing , 2014, 2014 IEEE Conference on Computational Intelligence and Games.

[4]  Mike Preuss,et al.  MCTS/EA hybrid GVGAI players and game difficulty estimation , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[5]  Sushil J. Louis,et al.  Using a genetic algorithm to tune first-person shooter bots , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[6]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[7]  Rémi Coulom,et al.  Computing "Elo Ratings" of Move Patterns in the Game of Go , 2007, J. Int. Comput. Games Assoc..

[8]  Moshe Sipper,et al.  EvoMCTS: A Scalable Approach for General Game Learning , 2014, IEEE Transactions on Computational Intelligence and AI in Games.

[9]  Michael Buro,et al.  Portfolio greedy search and simulation for large-scale combat in starcraft , 2013, 2013 IEEE Conference on Computational Inteligence in Games (CIG).

[10]  Rémi Coulom,et al.  Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search , 2006, Computers and Games.

[11]  H. Jaap van den Herik,et al.  Cross-Entropy for Monte-Carlo Tree Search , 2008, J. Int. Comput. Games Assoc..

[12]  Chiara F. Sironi,et al.  On-Line Parameter Tuning for Monte-Carlo Tree Search in General Game Playing , 2017, CGW@IJCAI.

[13]  Sylvain Gelly,et al.  Exploration exploitation in Go: UCT for Monte-Carlo Go , 2006, NIPS 2006.

[14]  Sebastian Risi,et al.  Continual online evolutionary planning for in-game build order adaptation in StarCraft , 2017, GECCO.

[15]  Simon M. Lucas,et al.  Analysis of Vanilla Rolling Horizon Evolution Parameters in General Video Game Playing , 2017, EvoApplications.

[16]  Simon M. Lucas,et al.  Population seeding techniques for Rolling Horizon Evolution in General Video Game Playing , 2017, 2017 IEEE Congress on Evolutionary Computation (CEC).

[17]  Csaba Szepesvári,et al.  Bandit Based Monte-Carlo Planning , 2006, ECML.

[18]  S.M. Lucas,et al.  Evolutionary computation and games , 2006, IEEE Computational Intelligence Magazine.

[19]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[20]  Simon M. Lucas,et al.  Fast Evolutionary Adaptation for Monte Carlo Tree Search , 2014, EvoApplications.

[21]  Simon M. Lucas,et al.  Rolling horizon evolution versus tree search for navigation in single-player real-time games , 2013, GECCO '13.

[22]  Michael Buro,et al.  Hierarchical Portfolio Search: Prismata's Robust AI Architecture for Games with Large Search Spaces , 2015, AIIDE.

[23]  Santiago Ontañón,et al.  The Combinatorial Multi-Armed Bandit Problem and Its Application to Real-Time Strategy Games , 2013, AIIDE.

[24]  Hans-Paul Schwefel,et al.  Evolution strategies – A comprehensive introduction , 2002, Natural Computing.

[25]  Peter I. Cowling,et al.  Ensemble Determinization in Monte Carlo Tree Search for the Imperfect Information Card Game Magic: The Gathering , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[26]  Tomáš Kozelek,et al.  Methods of MCTS and the game Arimaa , 2009 .

[27]  Tzung-Pei Hong,et al.  Adversarial Search by Evolutionary Computation , 2001, Evolutionary Computation.

[28]  H. Jaap van den Herik,et al.  Progressive Strategies for Monte-Carlo Tree Search , 2008 .

[29]  Michael H. Bowling,et al.  Monte Carlo Tree Search in Continuous Action Spaces with Execution Uncertainty , 2016, IJCAI.

[30]  Jos W. H. M. Uiterwijk,et al.  Single-player Monte-Carlo tree search for SameGame , 2012, Knowl. Based Syst..

[31]  Julian Togelius,et al.  Neuroevolution in Games: State of the Art and Open Challenges , 2014, IEEE Transactions on Computational Intelligence and AI in Games.

[32]  Demis Hassabis,et al.  Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm , 2017, ArXiv.