Exploring Adaptive MCTS with TD Learning in miniXCOM

In recent years, Monte Carlo tree search (MCTS) has achieved widespread adoption within the game community. Its use in conjunction with deep reinforcement learning has produced success stories in many applications. While these approaches have been implemented in various games, from simple board games to more complicated video games such as StarCraft, the use of deep neural networks requires a substantial training period. In this work, we explore on-line adaptivity in MCTS without requiring pre-training. We present MCTS-TD, an adaptive MCTS algorithm improved with temporal difference learning. We demonstrate our new approach on the game miniXCOM, a simplified version of XCOM, a popular commercial franchise consisting of several turn-based tactical games, and show how adaptivity in MCTS-TD allows for improved performances against opponents.

[1]  Jacek Ma'ndziuk,et al.  Monte Carlo Tree Search: a review of recent modifications and applications , 2021, Artificial Intelligence Review.

[2]  Raluca D. Gaina,et al.  TAG: Terraforming Mars , 2021, AIIDE.

[3]  P. González-Calero,et al.  Reinforcement Learning Methods to Evaluate the Impact of AI Changes in Game Design , 2021, AIIDE.

[4]  Levi H. S. Lelis,et al.  Birds in Boots: Learning to Play Angry Birds with Policy-Guided Search , 2021, AIIDE.

[5]  Matthew Guzdial,et al.  Explaining Deep Reinforcement Learning Agents In The Atari Domain through a Surrogate Model , 2021, AIIDE.

[6]  Cewu Lu,et al.  Combinatorial Q-Learning for Dou Di Zhu , 2020, Artificial Intelligence and Interactive Digital Entertainment Conference.

[7]  Julian Togelius,et al.  Tree Search versus Optimization Approaches for Map Generation , 2020, AIIDE.

[8]  Markus Eger,et al.  PAIndemic: A Planning Agent for Pandemic , 2020, AIIDE.

[9]  J. Togelius,et al.  PCGRL: Procedural Content Generation via Reinforcement Learning , 2020, AIIDE.

[10]  Matthew E. Taylor,et al.  Action Guidance with MCTS for Deep Reinforcement Learning , 2019, AIIDE.

[11]  S. Livingston,et al.  The Future Impact of Artificial Intelligence on Humans and Human Rights , 2019, Ethics & International Affairs.

[12]  Spyridon Samothrakis,et al.  On Monte Carlo Tree Search and Reinforcement Learning , 2017, J. Artif. Intell. Res..

[13]  A. Sima Etaner-Uyar,et al.  Monte Carlo tree search with temporal-difference learning for general video game playing , 2017, 2017 IEEE Conference on Computational Intelligence and Games (CIG).

[14]  David Churchill,et al.  Comparison of Monte Carlo Tree Search Methods in the Imperfect Information Card Game Cribbage , 2017 .

[15]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[16]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[17]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[18]  Darryl Charles,et al.  Machine learning in digital games: a survey , 2008, Artificial Intelligence Review.

[19]  Pieter Spronck,et al.  Monte-Carlo Tree Search: A New Framework for Game AI , 2008, AIIDE.

[20]  Rémi Coulom,et al.  Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search , 2006, Computers and Games.

[21]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[22]  Gerald Tesauro,et al.  Temporal difference learning and TD-Gammon , 1995, CACM.