Model Checking Algorithms for CTMDPs

Continuous Stochastic Logic (CSL) can be interpreted over continuoustime Markov decision processes (CTMDPs) to specify quantitative properties of stochastic systems that allow some external control. Model checking CSL formulae over CTMDPs requires then the computation of optimal control strategies to prove or disprove a formula. The paper presents a conservative extension of CSL over CTMDPs--with rewards--and exploits established results for CTMDPs for model checking CSL. A new numerical approach based on uniformization is devised to compute time bounded reachability results for time dependent control strategies. Experimental evidence is given showing the efficiency of the approach.

[1]  Ronald A. Howard,et al.  Dynamic Programming and Markov Processes , 1960 .

[2]  Anders Martin-Löf,et al.  Optimal Control of a Continuous-Time Markov Chain with Periodic Transition Probabilities , 1967, Oper. Res..

[3]  B. L. Miller.  Finite state continuous time Markov decision processes with an infinite planning horizon , 1968 .

[4]  B. L. Miller.  Finite State Continuous Time Markov Decision Processes with a Finite Planning Horizon , 1968 .

[5]  Mark R. Lembersky.  On Maximal Rewards and $|varepsilon$-Optimal Policies in Continuous Time Markov Decision Chains , 1974 .

[6]  Robin Milner,et al.  On Observing Nondeterminism and Concurrency , 1980, ICALP.

[7]  Micha Yadin,et al.  Randomization Procedures in the Computation of Cumulative-Time Distributions over Discrete State Markov Processes , 1984, Oper. Res..

[8]  Donald Gross,et al.  The Randomization Technique as a Modeling Tool and Solution Procedure for Transient Markov Processes , 1984, Oper. Res..

[9]  C. A. Petri,et al.  Concurrency Theory , 1986, Advances in Petri Nets.

[10]  Peter W. Glynn,et al.  Computing Poisson probabilities , 1988, CACM.

[11]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[12]  Christel Baier,et al.  Approximate Symbolic Model Checking of Continuous-Time Markov Chains , 1999, CONCUR.

[13]  Christel Baier,et al.  On the Logical Characterisation of Performability Properties , 2000, ICALP.

[14]  Robert K. Brayton,et al.  Model-checking continuous-time Markov chains , 2000, TOCL.

[15]  Christel Baier,et al.  Model-Checking Algorithms for Continuous-Time Markov Chains , 2002, IEEE Trans. Software Eng..

[16]  Christel Baier,et al.  Efficient Computation of Time-Bounded Reachability Probabilities in Uniform Continuous-Time Markov Decision Processes , 2005, TACAS.

[17]  Rajeev Alur,et al.  A Temporal Logic of Nested Calls and Returns , 2004, TACAS.

[18]  Ian Stark,et al.  Free-Algebra Models for the pi-Calculus , 2005, FoSSaCS.

[19]  Bernd Becker,et al.  Compositional Performability Evaluation for STATEMATE , 2006, Third International Conference on the Quantitative Evaluation of Systems - (QEST'06).

[20]  Joost-Pieter Katoen,et al.  Three-Valued Abstraction for Continuous-Time Markov Chains , 2007, CAV.

[21]  Joost-Pieter Katoen,et al.  The Ins and Outs of the Probabilistic Model Checker MRMC , 2009, 2009 Sixth International Conference on the Quantitative Evaluation of Systems.

[22]  Joost-Pieter Katoen,et al.  Delayed Nondeterminism in Continuous-Time Markov Decision Processes , 2009, FoSSaCS.

[23]  J. Katoen,et al.  Computing Maximum Reachability Probabilities in Markovian Timed Automata , 2010 .

[24]  Lijun Zhang,et al.  Time-Bounded Reachability Probabilities in Continuous-Time Markov Decision Processes , 2010, 2010 Seventh International Conference on the Quantitative Evaluation of Systems.

[25]  Lijun Zhang,et al.  Model Checking Interactive Markov Chains , 2010, TACAS.

[26]  Sven Schewe,et al.  Finite optimal control for time-bounded reachability in CTMDPs and continuous-time Markov games , 2010, Acta Informatica.

[27]  Peter Buchholz,et al.  Numerical analysis of continuous time Markov decision processes over finite horizons , 2011, Comput. Oper. Res..

[28]  Joost-Pieter Katoen,et al.  Reachability probabilities in Markovian Timed Automata , 2011, IEEE Conference on Decision and Control and European Control Conference.

[29]  Jan Kretínský,et al.  Continuous-Time Stochastic Games with Time-Bounded Reachability , 2013, FSTTCS.