The complexity of reachability in parametric Markov decision processes

Abstract This article presents the complexity of reachability decision problems for parametric Markov decision processes (pMDPs), an extension to Markov decision processes (MDPs) where transitions probabilities are described by polynomials over a finite set of parameters. In particular, we study the complexity of finding values for these parameters such that the induced MDP satisfies some maximal or minimal reachability probability constraints. We discuss different variants depending on the comparison operator in the constraints and the domain of the parameter values. We improve all known lower bounds for this problem, and notably provide ETR-completeness results for distinct variants of this problem.

[1]  Sebastian Junges,et al.  Are Parametric Markov Chains Monotonic? , 2019, ATVA.

[2]  Anne Condon,et al.  Computational models of games , 1989, ACM distinguished dissertations.

[3]  Marta Z. Kwiatkowska,et al.  PRISM 4.0: Verification of Probabilistic Real-Time Systems , 2011, CAV.

[4]  Andrew Chi-Chih Yao,et al.  The complexity of nonuniform random number generation , 1976 .

[5]  John F. Canny,et al.  Some algebraic and geometric computations in PSPACE , 1988, STOC '88.

[6]  Krishnendu Chatterjee,et al.  Robustness of Structurally Equivalent Concurrent Parity Games , 2011, FoSSaCS.

[7]  Christel Baier,et al.  Parametric Markov Chains: PCTL Complexity and Fraction-free Gaussian Elimination , 2017, GandALF.

[8]  Lubos Brim,et al.  Precise parameter synthesis for stochastic biochemical systems , 2014, Acta Informatica.

[9]  Olle Häggström.  Finite Markov Chains and Algorithmic Applications , 2002 .

[10]  Pedro R. D'Argenio,et al.  Distributed probabilistic input/output automata: Expressiveness, (un)decidability and algorithms , 2014, Theor. Comput. Sci..

[11]  U. Rieder,et al.  Markov Decision Processes , 2010 .

[12]  Lijun Zhang,et al.  Model Repair for Markov Decision Processes , 2013, 2013 International Symposium on Theoretical Aspects of Software Engineering.

[13]  Marcus Schaefer,et al.  Fixed Points, Nash Equilibria, and the Existential Theory of the Reals , 2017, Theory of Computing Systems.

[14]  Sebastian Junges,et al.  Synthesis in pMDPs: A Tale of 1001 Parameters , 2018, ATVA.

[15]  Sebastian Junges,et al.  Alternating Weak Automata from Universal Trees , 2019, CONCUR.

[16]  C. R. Ramakrishnan,et al.  Model Repair for Probabilistic Systems , 2011, TACAS.

[17]  Mahesh Viswanathan,et al.  Model-Checking Markov Chains in the Presence of Uncertainties , 2006, TACAS.

[18]  Sebastian Junges,et al.  A Storm is Coming: A Modern Probabilistic Model Checker , 2017, CAV.

[19]  Sebastian Junges,et al.  Shepherding Hordes of Markov Chains , 2019, TACAS.

[20]  Di Wu,et al.  Reachability analysis of uncertain systems using bounded-parameter Markov decision processes , 2008, Artif. Intell..

[21]  Christel Baier,et al.  Parametric Markov chains: PCTL complexity and fraction-free Gaussian elimination , 2020, Inf. Comput..

[22]  Sven Schewe,et al.  Accelerated Model Checking of Parametric Markov Chains , 2018, ATVA.

[23]  Neil Immerman,et al.  The Complexity of Decentralized Control of Markov Decision Processes , 2000, UAI.

[24]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[25]  Nils Jansen,et al.  Accelerating Parametric Probabilistic Verification , 2014, QEST.

[26]  Eilon Solan.  Continuity of the Value of Competitive Markov Decision Processes , 2003 .

[27]  Lijun Zhang,et al.  Synthesis for PCTL in Parametric Markov Decision Processes , 2011, NASA Formal Methods.

[28]  Scott Sanner,et al.  Solutions to Factored MDPs with Imprecise Transition Probabilities 1 , 2011 .

[29]  Krishnendu Chatterjee,et al.  A Symbolic SAT-Based Algorithm for Almost-Sure Reachability with Small Strategies in POMDPs , 2015, AAAI.

[30]  Shlomo Zilberstein,et al.  Formal models and algorithms for decentralized decision making under uncertainty , 2008, Autonomous Agents and Multi-Agent Systems.

[31]  Alberto L. Sangiovanni-Vincentelli,et al.  Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties , 2013, CAV.

[32]  Taolue Chen,et al.  On the complexity of model checking interval-valued discrete time Markov chains , 2013, Inf. Process. Lett..

[33]  Conrado Daws.  Symbolic and Parametric Model Checking of Discrete-Time Markov Chains , 2004, ICTAC.

[34]  Kim Guldstrand Larsen,et al.  Specification and refinement of probabilistic processes , 1991, [1991] Proceedings Sixth Annual IEEE Symposium on Logic in Computer Science.

[35]  James Renegar,et al.  On the Computational Complexity and Geometry of the First-Order Theory of the Reals, Part I: Introduction. Preliminaries. The Geometry of Semi-Algebraic Sets. The Decision Problem for the Existential Theory of the Reals , 1992, J. Symb. Comput..

[36]  Sebastian Junges,et al.  Parameter Synthesis for Markov Models: Faster Than Ever , 2016, ATVA.

[37]  David Barber,et al.  On the Computational Complexity of Stochastic Controller Optimization in POMDPs , 2011, TOCT.

[38]  Lijun Zhang,et al.  Probabilistic reachability for parametric Markov models , 2010, International Journal on Software Tools for Technology Transfer.

[39]  Sebastian Junges,et al.  Parameter Synthesis for Markov Models , 2019, Formal Methods Syst. Des..

[40]  Christel Baier,et al.  Principles of model checking , 2008 .

[41]  Luca Bortolussi,et al.  Bayesian Statistical Parameter Synthesis for Linear Temporal Properties of Stochastic Models , 2018, TACAS.

[42]  Carlo Ghezzi,et al.  Supporting Self-Adaptation via Quantitative Verification and Sensitivity Analysis at Run Time , 2016, IEEE Transactions on Software Engineering.

[43]  Andrea Maggiolo-Schettini,et al.  Parametric probabilistic transition systems for system design and analysis , 2007, Formal Aspects of Computing.

[44]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[45]  Sebastian Junges,et al.  Permissive Finite-State Controllers of POMDPs using Parameter Synthesis , 2017, ArXiv.

[46]  Sebastian Junges,et al.  Counterexample-Driven Synthesis for Probabilistic Program Sketches , 2019, FM.

[47]  Robert Givan,et al.  Bounded-parameter Markov decision processes , 2000, Artif. Intell..

[48]  Marcus Schaefer,et al.  Realizability of Graphs and Linkages , 2013 .

[49]  Ventsislav Chonev,et al.  Reachability in Augmented Interval Markov Chains , 2017, RP.

[50]  Jeremy Sproston.  Qualitative Reachability for Open Interval Markov Chains , 2018, RP.

[51]  David S. Rosenblum,et al.  Perturbation Analysis in Verification of Discrete-Time Markov Chains , 2014, CONCUR.