Greatest fixed points of probabilistic min/max polynomial equations, and reachability for branching Markov decision processes

Abstract We give polynomial time algorithms for quantitative (and qualitative) reachability analysis for Branching Markov Decision Processes (BMDPs). Specifically, given a BMDP, and given an initial population, where the objective of the controller is to maximize (or minimize) the probability of eventually reaching a population that contains an object of a desired (or undesired) type, we give algorithms for approximating the supremum (infimum) reachability probability, within desired precision ϵ > 0 , in time polynomial in the encoding size of the BMDP and in log ⁡ ( 1 / ϵ ) . We furthermore give P-time algorithms for computing ϵ-optimal strategies for both maximization and minimization of reachability probabilities. We also give P-time algorithms for all associated qualitative analysis problems, namely: deciding whether the optimal (supremum or infimum) reachability probabilities are 0 or 1. Prior to this paper, approximation of optimal reachability probabilities for BMDPs was not even known to be decidable. Our algorithms exploit the following basic fact: we show that for any BMDP, its maximum (minimum) non-reachability probabilities are given by the greatest fixed point (GFP) solution g ⁎ ∈ [ 0 , 1 ] n of a corresponding monotone max (min) Probabilistic Polynomial System of equations (max/minPPS), x = P ( x ) , which are the Bellman optimality equations for a BMDP with non-reachability objectives. We show how to compute the GFP of max/minPPSs to desired precision in P-time. We also study more general branching simple stochastic games (BSSGs) with (non-)reachability objectives. We show that: (1) the value of these games is captured by the GFP, g ⁎ , of a corresponding max-minPPS, x = P ( x ) ; (2) the quantitative problem of approximating the value is in TFNP; and (3) the qualitative problems associated with the value are all solvable in P-time.

[1]  Kousha Etessami,et al.  Recursive Markov chains, stochastic grammars, and monotone systems of nonlinear equations , 2005, JACM.

[2]  Javier Esparza,et al.  Model checking probabilistic pushdown automata , 2004, Proceedings of the 19th Annual IEEE Symposium on Logic in Computer Science, 2004..

[3]  E. Denardo,et al.  Totally expanding multiplicative systems , 2005 .

[4]  T. E. Harris,et al.  The Theory of Branching Processes. , 1963 .

[5]  Krishnendu Chatterjee,et al.  Evolutionary dynamics of cancer in response to targeted combination therapy , 2013, eLife.

[6]  Kousha Etessami,et al.  Recursive Stochastic Games with Positive Rewards , 2008, ICALP.

[7]  Tomás Brázdil,et al.  Reachability in recursive Markov decision processes , 2008, Inf. Comput..

[8]  Krishnendu Chatterjee,et al.  Efficient and Dynamic Algorithms for Alternating Büchi Games and Maximal End-Component Decomposition , 2014, J. ACM.

[9]  Natalia L. Komarova,et al.  Cancer: Calculated treatment , 2013, Nature.

[10]  Helmut Seidl,et al.  Approximative Methods for Monotone Systems of Min-Max-Polynomial Equations , 2008, ICALP.

[11]  Krishnendu Chatterjee,et al.  TTP: Tool for Tumor Progression , 2013, CAV.

[12]  Stefan Kiefer,et al.  Model Checking Stochastic Branching Processes , 2012, MFCS.

[13]  Kousha Etessami,et al.  Polynomial Time Algorithms for Multi-Type Branching Processes and Stochastic Context-Free Grammars , 2012, ArXiv.

[14]  Kousha Etessami,et al.  Recursive Markov Decision Processes and Recursive Stochastic Games , 2005, ICALP.

[15]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[16]  Peter Whittle,et al.  Growth Optimality for Branching Markov Decision Chains , 1982, Math. Oper. Res..

[17]  Tomás Brázdil,et al.  Qualitative reachability in stochastic BPA games , 2011, Inf. Comput..

[18]  S. Pliska.  Optimization of Multitype Branching Processes , 1976 .

[19]  Stefan Kiefer,et al.  Analysis of Probabilistic Basic Parallel Processes , 2014, FoSSaCS.

[20]  Kousha Etessami,et al.  Polynomial Time Algorithms for Branching Markov Decision Processes and Probabilistic Min(Max) Polynomial Bellman Equations , 2012, ICALP.

[21]  HenzingerMonika,et al.  Efficient and Dynamic Algorithms for Alternating Büchi Games and Maximal End-Component Decomposition , 2014 .