Decentralized No-regret Learning Algorithms for Extensive-form Correlated Equilibria (Extended Abstract)

The existence of uncoupled no-regret learning dynamics converging to correlated equilibria in normal-form games is a celebrated result in the theory of multi-agent systems. Specifically, it has been known for more than 20 years that when all players seek to minimize their internal regret in a repeated normal-form game, the empirical frequency of play converges to a normal-form correlated equilibrium. Extensive-form games generalize normal-form games by modeling both sequential and simultaneous moves, as well as imperfect information. Because of the sequential nature and the presence of private information, correlation in extensive-form games possesses significantly different properties than in normal-form games. The extensive-form correlated equilibrium (EFCE) is the natural extensive-form counterpart to the classical notion of correlated equilibrium in normal-form games. Compared to the latter, the constraints that define the set of EFCEs are significantly more complex, as the correlation device (a.k.a. mediator) must take into account the evolution of beliefs of each player as they make observations throughout the game. Due to this additional complexity, the existence of uncoupled learning dynamics leading to an EFCE has remained a challenging open research question for a long time. In this article, we settle that question by giving the first uncoupled noregret dynamics which provably converge to the set of EFCEs in n-player general-sum extensive-form games with perfect recall. We show that each iterate can be computed in time polynomial in the size of the game tree, and that, when all players play repeatedly according to our learning dynamics, the empirical frequency of play after T game repetitions is guaranteed to be a O(1/ √ T )-approximate EFCE with high probability, and an EFCE almost surely in the limit. ∗The complete version of this paper won a best paper award at NeurIPS 2020 [Celli et al., 2020]. Some of the results presented here only appear in the full version of the paper [Farina et al., 2021]. †Contact Author. 1 Motivation This work studies decision-making problems in which rational individuals interact with a centralized planner. The centralized planner cannot directly tell the individuals what to do, but the goal of the centralized planner is to steer the individuals’ behaviors to mutually beneficial outcomes. There are many real-world problems where we observe this type of interaction, and this is increasingly common in the gig economy we all live in today. Think, for example, of ride-sharing or food delivery platforms, where drivers provide services to customers, and the whole market is centralized through a single app that every agent connects to. Because the individual decision makers in the system have free will, the central planner has to take into consideration the fact that all of the individual decision makers will act selfishly according to their objectives. Therefore, to get them to behave in a certain way, the central planner must nudge them using the right incentives. This type of soft coordination is already enough to steer the system to social welfare that would be largely impossible in absence of a central planner, so without any form of coordination between the decision makers. The strategy that the central planner should follow when interacting with decision makers is called a correlated equilibrium in the game theory literature. The key feature of a correlated equilibrium is that all the decision makers receive the right incentives to follow the planner’s recommendations. This means that no agent would want to do something different from what they are recommended to do by the central planner. The study of this type of equilibrium goes back to the seminal work on correlated equilibrium by Robert Aumann in 1974 [Aumann, 1974], who was later awarded a Nobel prize in economics for his work on game-theoretic cooperation. Since then, there has been much effort in scaling up the computation of correlated equilibria and designing algorithms guaranteeing some key properties. In particular, a crucial property that algorithms for computing correlated equilibria should satisfy is decentralization. This essentially means that the behavior and incentives of each agent should be computed independently from the other agents. The decentralization is fundamental for the computation to scale well, and it allows the agents to converge to an equilibrium point without the need for a central planner. Furthermore, decentralization preserves the agents’ privacy during the learning process. Indeed, agents should not need to report their Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21) Sister Conferences Best Papers Track