Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties

We address the problem of verifying Probabilistic Computation Tree Logic (PCTL) properties of Markov Decision Processes (MDPs) whose state transition probabilities are only known to lie within uncertainty sets. We first introduce the model of Convex-MDPs (CMDPs), i.e., MDPs with convex uncertainty sets. CMDPs generalize Interval-MDPs (IMDPs) by allowing also more expressive (convex) descriptions of uncertainty. Using results on strong duality for convex programs, we then present a PCTL verification algorithm for CMDPs, and prove that it runs in time polynomial in the size of a CMDP for a rich subclass of convex uncertainty models. This result allows us to lower the previously known algorithmic complexity upper bound for IMDPs from co-NP to PTIME. We demonstrate the practical effectiveness of the proposed approach by verifying a consensus protocol and a dynamic configuration protocol for IPv4 addresses.

[1]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[2]  J. Blass,et al.  Symposium , 1979, The Lancet.

[3]  Edmund M. Clarke,et al.  Design and Synthesis of Synchronization Skeletons Using Branching-Time Temporal Logic , 1981, Logic of Programs.

[4]  Moshe Y. Vardi.  Automatic verification of probabilistic concurrent finite state programs , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[5]  Maurice Herlihy,et al.  Fast Randomized Consensus Using Shared Memory , 1990, J. Algorithms.

[6]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[7]  Yurii Nesterov,et al.  Interior-point polynomial algorithms in convex programming , 1994, Siam studies in applied mathematics.

[8]  Andrea Bianco,et al.  Model Checking of Probabalistic and Nondeterministic Systems , 1995, FSTTCS.

[9]  Mihalis Yannakakis,et al.  The complexity of probabilistic verification , 1995, JACM.

[10]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[11]  Arkadi Nemirovski,et al.  Robust solutions of uncertain linear programs , 1999, Oper. Res. Lett..

[12]  Andrew McCallum,et al.  Maximum Entropy Markov Models for Information Extraction and Segmentation , 2000, ICML.

[13]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[14]  Marta Z. Kwiatkowska,et al.  Automated Verification of a Randomized Distributed Consensus Protocol Using Cadence SMV and PRISM , 2001, CAV.

[15]  Lev V. Utkin,et al.  Interval-Valued Finite Markov Chains , 2002, Reliab. Comput..

[16]  Holger Hermanns,et al.  ETMCC: model checking performability properties of Markov chains , 2003, 2003 International Conference on Dependable Systems and Networks, 2003. Proceedings..

[17]  Marta Z. Kwiatkowska,et al.  Performance analysis of probabilistic timed automata using digital clocks , 2003, Formal Methods Syst. Des..

[18]  Mahesh Viswanathan,et al.  VESTA: A statistical model-checker and analyzer for probabilistic systems , 2005, Second International Conference on the Quantitative Evaluation of Systems (QEST'05).

[19]  Håkan L. S. Younes.  Ymer: A Statistical Model Checker , 2005, CAV.

[20]  Bengt Jonsson,et al.  A logic for reasoning about time and reliability , 1990, Formal Aspects of Computing.

[21]  Laurent El Ghaoui,et al.  Robust Control of Markov Decision Processes with Uncertain Transition Matrices , 2005, Oper. Res..

[22]  J. Gill.  An entropy measure of uncertainty in vote choice , 2005 .

[23]  Stuart Cheshire,et al.  Dynamic Configuration of IPv4 Link-Local Addresses , 2005, RFC.

[24]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[25]  Mahesh Viswanathan,et al.  Model-Checking Markov Chains in the Presence of Uncertainties , 2006, TACAS.

[26]  Quantitative verification: models techniques and tools , 2007, ESEC-FSE '07.

[27]  T. Henzinger,et al.  Model-Checking ω-Regular Properties of Interval Markov Chains , 2008 .

[28]  A. Neumaier,et al.  Towards a Combination of Interval and Ellipsoid Uncertainty , 2008 .

[29]  Lijun Zhang,et al.  INFAMY: An Infinite-State Markov Model Checker , 2009, CAV.

[30]  Joost-Pieter Katoen,et al.  The Ins and Outs of the Probabilistic Model Checker MRMC , 2009, 2009 Sixth International Conference on the Quantitative Evaluation of Systems.

[31]  Lijun Zhang,et al.  Synthesis for PCTL in Parametric Markov Decision Processes , 2011, NASA Formal Methods.

[32]  Verena Wolf,et al.  Parameter Identification for Markov Models of Biochemical Reactions , 2011, CAV.

[33]  Marta Z. Kwiatkowska,et al.  PRISM 4.0: Verification of Probabilistic Real-Time Systems , 2011, CAV.

[34]  Marta Z. Kwiatkowska,et al.  Automated Verification Techniques for Probabilistic Systems , 2011, SFM.

[35]  C. Belta,et al.  Control of Markov decision processes from PCTL specifications , 2011, Proceedings of the 2011 American Control Conference.

[36]  Joost-Pieter Katoen,et al.  Robust PCTL model checking , 2012, HSCC '12.

[37]  Calin Belta,et al.  Temporal Logic Motion Planning and Control With Probabilistic Satisfaction Guarantees , 2012, IEEE Transactions on Robotics.

[38]  Ufuk Topcu,et al.  Robust control of uncertain Markov Decision Processes with temporal logic specifications , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[39]  Paolo Milazzo,et al.  Probabilistic model checking of biological systems with uncertain kinetic rates , 2012, Theor. Comput. Sci..