Adaptive Playouts in Monte-Carlo Tree Search with Policy-Gradient Reinforcement Learning

Monte-Carlo Tree Search evaluates positions with the help of a playout policy. If the playout policy evaluates a position wrong then there are cases where the tree-search has difficulties to find the correct move due to the large search-space. This paper explores adaptive playout-policies which improve the playout-policy during a tree-search. With the help of policy-gradient reinforcement learning techniques we optimize the playout-policy to give better evaluations. We tested the algorithm in Computer Go and measured an increase in playing strength of more than 100 ELO. The resulting program was able to deal with difficult test-cases which are known to pose a problem for Monte-Carlo-Tree-Search.

[1]  Marco Platzner,et al.  Common fate graph patterns in Monte Carlo Tree Search for computer go , 2014, 2014 IEEE Conference on Computational Intelligence and Games.

[2]  Richard S. Sutton,et al.  Sample-based learning and search with permanent and transient memories , 2008, ICML '08.

[3]  Simon M. Lucas,et al.  Knowledge-based fast evolutionary MCTS for general video game playing , 2014, 2014 IEEE Conference on Computational Intelligence and Games.

[4]  Kokolo Ikeda,et al.  Efficiency of Static Knowledge Bias in Monte-Carlo Tree Search , 2013, Computers and Games.

[5]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[6]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[7]  Shih-Chieh Huang,et al.  Investigating the Limits of Monte-Carlo Tree Search Methods in Computer Go , 2013, Computers and Games.

[8]  Simon M. Lucas,et al.  Fast Evolutionary Adaptation for Monte Carlo Tree Search , 2014, EvoApplications.

[9]  Shih-Chieh Huang,et al.  Monte-Carlo Simulation Balancing in Practice , 2010, Computers and Games.

[10]  Hendrik Baier,et al.  Adaptive Playout Policies for Monte-Carlo Go , 2010 .

[11]  H. Jaap van den Herik,et al.  Progressive Strategies for Monte-Carlo Tree Search , 2008 .

[12]  David Silver,et al.  Reinforcement learning and simulation-based search in computer go , 2009 .

[13]  Hendrik Baier,et al.  The Power of Forgetting: Improving the Last-Good-Reply Policy in Monte Carlo Go , 2010, IEEE Transactions on Computational Intelligence and AI in Games.

[14]  Petr Baudis,et al.  PACHI: State of the Art Open Source Go Program , 2011, ACG.

[15]  David Silver,et al.  Combining online and offline knowledge in UCT , 2007, ICML '07.