Enhancing Monte Carlo Tree Search for Playing Hearthstone

Hearthstone is a popular online collectible card game (CCG). Hearthstone imposes interesting challenges in developing a search algorithm for the game AI. As a CCG, it has a considerable amount of hidden information from each player’s private hand and deck. Moreover, the action space is full of stochastic actions compared to other similar games. That is, instead of a single move, each player is allowed to build a move sequence via various combinations of atomic actions. Therefore, when applying any heuristic search algorithm, the branching factor of the search space is extremely large. In this paper, we explore the use of Monte Carlo Tree Search (MCTS) with approaches to reduce the complexity of the search space and decide on the best strategy. First, we utilise state abstraction to present the search space as a Directed Acyclic Graph (DAG) and introduce a variant of Upper Confidence Bound for Trees (UCT) algorithm for the DAG. Next, we apply the sparse sampling algorithm to handle imperfect information and randomness and reduce the stochastic branching factor. This paper presents empirical evaluations of the proposed framework for Hearthstone and the experimental results suggest that our approach is well suited for developing a better AI agent.

[1]  H. Jaap van den Herik,et al.  Parallel Monte-Carlo Tree Search , 2008, Computers and Games.

[2]  David Silver,et al.  Smooth UCT Search in Computer Poker , 2015, IJCAI.

[3]  Zhengxing Chen,et al.  Q-DeckRec: A Fast Deck Recommendation System for Collectible Card Games , 2018, 2018 IEEE Conference on Computational Intelligence and Games (CIG).

[4]  Michael Buro,et al.  Improving hearthstone AI by learning high-level rollout policies and bucketing chance node events , 2017, 2017 IEEE Conference on Computational Intelligence and Games (CIG).

[5]  Hugo Morais,et al.  HoningStone: Building Creative Combos With Honing Theory for a Digital Card Game , 2017, IEEE Transactions on Computational Intelligence and AI in Games.

[6]  Andrzej Janusz,et al.  Improving Hearthstone AI by Combining MCTS and Supervised Learning Algorithms , 2018, 2018 IEEE Conference on Computational Intelligence and Games (CIG).

[7]  Julian Togelius,et al.  Exploring the hearthstone deck space , 2018, FDG.

[8]  Yishay Mansour,et al.  A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes , 1999, Machine Learning.

[9]  Nataliya Sokolovska,et al.  Continuous Upper Confidence Trees , 2011, LION.

[10]  Vladimir Ulyantsev,et al.  Applying Reinforcement Learning and Supervised Learning Techniques to Play Hearthstone , 2017, 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA).

[11]  Alexander Dockhorn,et al.  Introducing the Hearthstone-AI Competition , 2019, ArXiv.

[12]  Csaba Szepesvári,et al.  Bandit Based Monte-Carlo Planning , 2006, ECML.

[13]  Francisco S. Melo,et al.  Monte Carlo tree search experiments in hearthstone , 2017, 2017 IEEE Conference on Computational Intelligence and Games (CIG).

[14]  Peter I. Cowling,et al.  Evolutionary MCTS for Multi-Action Adversarial Games , 2018, 2018 IEEE Conference on Computational Intelligence and Games (CIG).

[15]  Peter I. Cowling,et al.  Information Set Monte Carlo Tree Search , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[16]  Feng Xiao,et al.  Pruning in UCT Algorithm , 2010, 2010 International Conference on Technologies and Applications of Artificial Intelligence.

[17]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[18]  Abdallah Saffidine,et al.  UCD: Upper Confidence Bound for Rooted Directed Acyclic Graphs , 2010 .

[19]  Alan Fern,et al.  Lower Bounding Klondike Solitaire with Monte-Carlo Planning , 2009, ICAPS.

[20]  Joel Veness,et al.  Monte-Carlo Planning in Large POMDPs , 2010, NIPS.

[21]  Levente Kocsis,et al.  Transpositions and move groups in Monte Carlo tree search , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[22]  Nathan R. Sturtevant,et al.  Understanding the Success of Perfect Information Monte Carlo Sampling in Game Tree Search , 2010, AAAI.

[23]  Johannes Maucher,et al.  Hearthstone deck-construction with a utility system , 2016, 2016 10th International Conference on Software, Knowledge, Information Management & Applications (SKIMA).

[24]  Peter I. Cowling,et al.  Ensemble Determinization in Monte Carlo Tree Search for the Imperfect Information Card Game Magic: The Gathering , 2012, IEEE Transactions on Computational Intelligence and AI in Games.