Solving for Best Responses and Equilibria in Extensive-Form Games with Reinforcement Learning Methods

We present a framework to solve for best responses and equilibria in extensive-form games (EFGs) of imperfect information by transforming a game together with an other-agent policy into a set of Markov decision processes (MDPs), one per player, and then applying simulation-based reinforcement learning (RL) to the ensuing MDPs. More specifically, we first transform a turn-taking partially observable Markov game (TT-POMG) into a set of partially observable Markov decision processes (POMDPs), and we then transform that set of POMDPs into a corresponding set of Markov decision processes (MDPs). Next, we observe that EFGs are a special case of TT-POMGs, and hence can be transformed as described. Furthermore, because each transformation preserves the strategically-relevant information of the model to which it is applied, an optimal policy in one of the ensuing MDPs corresponds to a best response to the given other-agent policy in the original EFG. We then go on to prove that our RL algorithm finds a near-optimal policy (and therefore a near-best response in the original EFG) in finite time, although the sample complexity is lower bounded by a function with an exponential dependence on the horizon. Nonetheless, we apply this algorithm iteratively to search for equilibria in an EFG. When the iterative procedure converges, the resulting MDP policies comprise an approximate weak perfect Bayesian equilibrium. Although this procedure is not guaranteed to converge, it frequently did in numerical experiments with sequential auctio.