LinUCB Applied to Monte-Carlo Tree Search

UCT is a de facto standard method for Monte-Carlo tree search (MCTS) algorithms, which have been applied to various domains and have achieved remarkable success. This study proposes a family of LinUCT algorithms that incorporate LinUCB into MCTS algorithms. LinUCB is a recently developed method that generalizes past episodes by ridge regression with feature vectors and rewards. LinUCB outperforms UCB1 in contextual multi-armed bandit problems. We introduce a straightforward application of LinUCB, \(\text {LinUCT}_{\text {PLAIN}}\) by substituting UCB1 with LinUCB in UCT. We show that it does not work well owing to the minimax structure of game trees. To better handle such tree structures, we present \(\text {LinUCT}_{\text {RAVE}}\) and \(\text {LinUCT}_{\text {FP}}\) by further incorporating two existing techniques, rapid action value estimation (RAVE) and feature propagation, which recursively propagates the feature vector of a node to that of its parent. Experiments were conducted with a synthetic model, which is an extension of the standard incremental random tree model in which each node has a feature vector that represents the characteristics of the corresponding position. The experimental results indicate that \(\text {LinUCT}_{\text {RAVE}}\), \(\text {LinUCT}_{\text {FP}}\), and their combination \(\text {LinUCT}_{\text {RAVE-FP}}\) outperform UCT, especially when the branching factor is relatively large.