Comparison of Algorithms for Simple Stochastic Games (Full Version)

Simple stochastic games are turn-based 2.5-player zero-sum graph games with a reachability objective. The problem is to compute the winning probability as well as the optimal strategies of both players. In this paper, we compare the three known classes of algorithms -- value iteration, strategy iteration and quadratic programming -- both theoretically and practically. Further, we suggest several improvements for all algorithms, including the first approach based on quadratic programming that avoids transforming the stochastic game to a stopping one. Our extensive experiments show that these improvements can lead to significant speed-ups. We implemented all algorithms in PRISM-games 3.0, thereby providing the first implementation of quadratic programming for solving simple stochastic games.

[1]  Krishnendu Chatterjee,et al.  Value Iteration , 2008, 25 Years of Model Checking.

[2]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[3]  Peter Bro Miltersen,et al.  Solving simple stochastic games with few coin toss positions , 2012, ESA.

[4]  Krishnendu Chatterjee,et al.  Verification of Markov Decision Processes Using Learning Algorithms , 2014, ATVA.

[5]  Krishnendu Chatterjee,et al.  Strategy improvement for concurrent reachability and turn-based stochastic safety games☆☆☆ , 2013, J. Comput. Syst. Sci..

[6]  Rafal Somla.  New Algorithms for Solving Simple Stochastic Games , 2005, Electron. Notes Theor. Comput. Sci..

[7]  Krishnendu Chatterjee,et al.  A survey of stochastic ω-regular games , 2012, J. Comput. Syst. Sci..

[8]  Benjamin Monmege,et al.  Interval iteration algorithm for MDPs and IMDPs , 2017, Theor. Comput. Sci..

[9]  Taolue Chen,et al.  On Stochastic Games with Multiple Objectives , 2013, MFCS.

[10]  Marta Z. Kwiatkowska,et al.  Quantitative verification and strategy synthesis for stochastic games , 2016, European Journal of Control.

[11]  Jan Kretínský,et al.  PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games , 2019, CAV.

[12]  Walter Ludwig,et al.  A Subexponential Randomized Algorithm for the Simple Stochastic Game Problem , 1995, Inf. Comput..

[13]  Tobias Meggendorfer,et al.  Satisfiability Bounds for ω-Regular Properties in Bounded-Parameter Markov Decision Processes , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[14]  Krishnendu Chatterjee,et al.  Stochastic Games with Lexicographic Reachability-Safety Objectives , 2020, CAV.

[15]  Jan Kretínský,et al.  Value Iteration for Simple Stochastic Games: Stopping Criterion and Learning Algorithm , 2018, CAV.

[16]  Christel Baier,et al.  Ensuring the Reliability of Your Model Checker: Interval Iteration for Markov Decision Processes , 2017, CAV.

[17]  Anne Condon,et al.  The Complexity of Stochastic Games , 1992, Inf. Comput..

[18]  Rong Ge,et al.  Another Sub-exponential Algorithm for the Simple Stochastic Game , 2010, Algorithmica.

[19]  Anne Condon,et al.  On Algorithms for Simple Stochastic Games , 1990, Advances In Computational Complexity Theory.

[20]  Mihalis Yannakakis,et al.  The complexity of probabilistic verification , 1995, JACM.

[21]  L. Khachiyan,et al.  The polynomial solvability of convex quadratic programming , 1980 .

[22]  Gabriel Santos,et al.  PRISM-games 3.0: Stochastic Game Verification with Concurrency, Equilibria and Time , 2020, CAV.

[23]  Christel Baier,et al.  Principles of model checking , 2008 .

[24]  Kristoffer Arnsfelt Hansen,et al.  The Complexity of Solving Reachability Games Using Value and Strategy Iteration , 2013, Theory of Computing Systems.

[25]  Krishnendu Chatterjee,et al.  Model-Checking omega-Regular Properties of Interval Markov Chains , 2008, FoSSaCS.

[26]  Krishnendu Chatterjee,et al.  A reduction from parity games to simple stochastic games , 2011, GandALF.

[27]  Jan Kretínský,et al.  Of Cores: A Partial-Exploration Framework for Markov Decision Processes , 2019, CONCUR.

[28]  Pranav Ashok,et al.  Approximating Values of Generalized-Reachability Stochastic Games , 2019, LICS.

[29]  R. Karp,et al.  On Nonterminating Stochastic Games , 1966 .

[30]  Peng Dai,et al.  Topological Value Iteration Algorithms , 2011, J. Artif. Intell. Res..

[31]  Jan Kretínský,et al.  Efficient Strategy Iteration for Mean Payoff in Markov Decision Processes , 2017, ATVA.

[32]  Joost-Pieter Katoen,et al.  Sound Value Iteration , 2018, CAV.

[33]  Florian Horn,et al.  Simple Stochastic Games with Few Random Vertices Are Easy to Solve , 2008, FoSSaCS.

[34]  Krishnendu Chatterjee,et al.  Gist: A Solver for Probabilistic Games , 2010, CAV.

[35]  Arnd Hartmanns,et al.  Optimistic Value Iteration , 2019, CAV.

[36]  Alois Knoll,et al.  GAVS+: An Open Platform for the Research of Algorithmic Game Solving , 2011, TACAS.