Finite State Machines Play Extensive-Form Games

Finite state machines are a well-known representation of strategies in (in)finitely repeated or stochastic games. Actions of players correspond to states in the machine and the transition between machine-states are caused by observations in the game. For extensive-form games (EFGs), machines can act as a formal grounding for abstraction methods used for solving large EFGs and as a domain-independent approach for generating sufficiently compact abstractions. We show that using machines of a restricted size in EFGs can both (i) reduce the theoretical complexity of computing some solution concepts, including Strong Stackelberg Equilibrium (SSE), (ii) as well as bring new practical algorithms that compute near-optimal equilibria considering only a fraction of strategy space. Our contributions include (1) formal definition and theoretical characterization of machine strategies in EFGs, (2) formal definitions and complexity analysis for solution concepts and their computation when restricted to small classes of machines, (3) new algorithms for computing SSE in general-sum games and Nash Equilibrium in zero-sum games that both directly use the concept of machines. Experimental results on two different domains show that the algorithms compute near-optimal strategies and achieve significantly better scalability compared to previous state-of-the-art algorithms.

[1]  Arie W. Kruglanski,et al.  Lay Epistemics and Human Knowledge: Cognitive and Motivational Bases , 2013 .

[2]  Troels Bjerre Lund,et al.  A heads-up no-limit Texas Hold'em poker player: discretized betting models and automatically generated equilibrium-finding programs , 2008, AAMAS.

[3]  A. Rubinstein,et al.  The Structure of Nash Equilibrium in Repeated Games with Finite Automata (Now published in Econometrica, 56 (1988), pp.1259-1282.) , 1986 .

[4]  Duane Szafron,et al.  Using Sliding Windows to Generate Action Abstractions in Extensive-Form Games , 2012, AAAI.

[5]  Lawrence M. Ausubel,et al.  Bargaining in Incomplete Information , 2002 .

[6]  Silvio Micali,et al.  On Play by Means of Computing Machines (Preliminary Version).@@@A Theory of Higher Order Probabilities.@@@Knowledge and Efficient Computation.@@@Realizability Semantics for Error-Tolerant Logics (Preliminary Version). , 1988 .

[7]  Tuomas Sandholm,et al.  Extensive-form game abstraction with bounds , 2014, EC.

[8]  Leonidas J. Guibas,et al.  A Visibility-Based Pursuit-Evasion Problem , 1999, Int. J. Comput. Geom. Appl..

[9]  Michael H. Bowling,et al.  Evaluating state-space abstractions in extensive-form games , 2013, AAMAS.

[10]  Jakub Černý,et al.  Stackelberg Extensive-Form Correlated Equilibrium with Multiple Followers , 2022 .

[11]  S. Chaiken.  The heuristic model of persuasion. , 1987 .

[12]  Jonathan Schaeffer,et al.  Approximating Game-Theoretic Optimal Strategies for Full-scale Poker , 2003, IJCAI.

[13]  Herbert S. Wilf,et al.  Generating functionology , 1990 .

[14]  N. Chater.  The Search for Simplicity: A Fundamental Cognitive Principle? , 1999 .

[15]  A. Rubinstein.  COMMENTS ON THE INTERPRETATION OF GAME THEORY , 1991 .

[16]  John H. Miller,et al.  The coevolution of automata in the repeated Prisoner's Dilemma , 1996 .

[17]  Juliane Hahn,et al.  Security And Game Theory Algorithms Deployed Systems Lessons Learned , 2016 .

[18]  P. Jean-Jacques Herings,et al.  Stationary Equilibria in Stochastic Games: Structure, Selection and Computation , 2000 .

[19]  Robert J. Aumann,et al.  Essays in game theory and mathematical economics in honor of Oskar Morgenstern , 1981 .

[20]  Branislav Bosanský,et al.  Computation of Stackelberg Equilibria of Finite Sequential Games , 2015, WINE.

[21]  E. Kalai.  Bounded Rationality and Strategic Complexity in Repeated Games , 1987 .

[22]  Javier Peña,et al.  Smoothing Techniques for Computing Nash Equilibria of Sequential Games , 2010, Math. Oper. Res..

[23]  B. Stengel,et al.  Efficient Computation of Behavior Strategies , 1996 .

[24]  Vincent Conitzer,et al.  Computing optimal strategies to commit to in extensive-form games , 2010, EC '10.

[25]  Tuomas Sandholm,et al.  Lossless abstraction of imperfect information games , 2007, JACM.

[26]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[27]  Branislav Bosanský,et al.  Using Correlated Strategies for Computing Stackelberg Equilibria in Extensive-Form Games , 2016, AAAI.

[28]  Giorgio Brajnik,et al.  Qualitative modeling and simulation of socio-economic phenomena , 1998, J. Artif. Soc. Soc. Simul..

[29]  Y. Ho,et al.  Differential games and optimal pursuit-evasion strategies , 1965 .

[30]  A. Neyman.  Bounded complexity justifies cooperation in the finitely repeated prisoners' dilemma , 1985 .

[31]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[32]  D. Koller,et al.  Efficient Computation of Equilibria for Extensive Two-Person Games , 1996 .

[33]  Kousha Etessami,et al.  The complexity of computing a (perfect) equilibrium for an n-player extensive form game of perfect recall , 2014, ArXiv.

[34]  Gábor Horváth,et al.  FlipThem: Modeling Targeted Attacks with FlipIt for Multiple Resources , 2014, GameSec.

[35]  J. Banks,et al.  Repeated games, finite automata, and complexity , 1990 .

[36]  Troels Bjerre Lund,et al.  Potential-Aware Automated Abstraction of Sequential Games, and Holistic Equilibrium Analysis of Texas Hold'em Poker , 2007, AAAI.

[37]  Branislav Bosanský,et al.  Sequence-Form Algorithm for Computing Stackelberg Equilibria in Extensive-Form Games , 2015, AAAI.

[38]  Branislav Bosanský,et al.  An Exact Double-Oracle Algorithm for Zero-Sum Extensive-Form Games with Imperfect Information , 2014, J. Artif. Intell. Res..

[39]  L. Samuelson,et al.  Evolutionary stability in repeated games played by finite automata , 1992 .

[40]  Tuomas Sandholm,et al.  Hierarchical Abstraction, Distributed Equilibrium Computation, and Post-Processing, with Application to a Champion No-Limit Texas Hold'em Agent , 2015, AAAI Workshop: Computer Poker and Imperfect Information.

[41]  S. Shankar Sastry,et al.  Probabilistic pursuit-evasion games: theory, implementation, and experimental evaluation , 2002, IEEE Trans. Robotics Autom..

[42]  Tuomas Sandholm,et al.  Potential-Aware Imperfect-Recall Abstraction with Earth Mover's Distance in Imperfect-Information Games , 2014, AAAI.

[43]  Joseph Y. Halpern,et al.  Computational Extensive-Form Games , 2015, EC.

[44]  A. Rubinstein.  Finite automata play the repeated prisoner's dilemma , 1986 .

[45]  Bernhard von Stengel,et al.  Extensive-Form Correlated Equilibrium: Definition and Computational Complexity , 2008, Math. Oper. Res..

[46]  Branislav Bosanský,et al.  An Initial Study of Targeted Personality Models in the FlipIt Game , 2018, GameSec.

[47]  Jacek Mandziuk,et al.  Double-oracle sampling method for Stackelberg Equilibrium approximation in general-sum extensive-form games , 2020, AAAI.

[48]  Ronald L. Rivest,et al.  FlipIt: The Game of “Stealthy Takeover” , 2012, Journal of Cryptology.

[49]  Michael L. Littman,et al.  Abstraction Methods for Game Theoretic Poker , 2000, Computers and Games.

[50]  David J. Chalmers,et al.  Does a rock implement every finite-state automaton? , 1996, Synthese.

[51]  Kousha Etessami,et al.  On the Complexity of Nash Equilibria and Other Fixed Points , 2010, SIAM J. Comput..

[52]  Michael H. Bowling,et al.  Solving Heads-Up Limit Texas Hold'em , 2015, IJCAI.

[53]  R. Aumann.  Survey of Repeated Games , 1981 .

[54]  Thomas C. Schelling,et al.  Dynamic models of segregation , 1971 .

[55]  Charles R. Schwenk,et al.  TOP MANAGEMENT, STRATEGY AND ORGANIZATIONAL KNOWLEDGE STRUCTURES , 1992 .

[56]  T. D. Parsons,et al.  Pursuit-evasion in a graph , 1978 .

[57]  Tuomas Sandholm,et al.  Imperfect-Recall Abstractions with Bounds in Games , 2014, EC.

[58]  Michael H. Bowling,et al.  No-Regret Learning in Extensive-Form Games with Imperfect Recall , 2012, ICML.

[59]  Michael H. Bowling,et al.  Regret Minimization in Games with Incomplete Information , 2007, NIPS.

[60]  H. Young,et al.  Handbook of Game Theory with Economic Applications , 2015 .

[61]  Tuomas Sandholm,et al.  Abstraction for Solving Large Incomplete-Information Games , 2015, AAAI.

[62]  Jakub Cerný,et al.  Incremental Strategy Generation for Stackelberg Equilibria in Extensive-Form Games , 2018, EC.

[63]  J. R. Pomerantz,et al.  THEORETICAL APPROACHES TO PERCEPTUAL ORGANIZATION Simplicity and Likelihood Principles , 1986 .

[64]  Robert Axelrod,et al.  Schema Theory: An Information Processing Model of Perception and Cognition , 1973, American Political Science Review.

[65]  Ronald L. Rivest,et al.  Defending against the Unknown Enemy: Applying FlipIt to System Security , 2012, GameSec.

[66]  A. Nerode,et al.  Linear automaton transformations , 1958 .

[67]  Duane Szafron,et al.  Automated Action Abstraction of Imperfect Information Extensive-Form Games , 2011, AAAI.

[68]  Andreas Flache,et al.  Understanding Complex Social Dynamics: A Plea For Cellular Automata Based Modelling , 1998, J. Artif. Soc. Soc. Simul..

[69]  Joshua M. Epstein,et al.  Growing Artificial Societies: Social Science from the Bottom Up , 1996 .

[70]  J. M. Sakoda.  The checkerboard model of social interaction , 1971 .

[71]  Nicolas Bonichon,et al.  Planar Graphs, via Well-Orderly Maps and Trees , 2004, WG.

[72]  Kevin Waugh,et al.  Theoretical and Practical Advances on Smoothing for Extensive-Form Games , 2017, EC.

[73]  Michael H. Bowling,et al.  Finding Optimal Abstract Strategies in Extensive-Form Games , 2012, AAAI.