Observing Continuous-Time MDPs by 1-Clock Timed Automata

This paper considers the verification of continuous-time Markov decision process (CTMDPs) against single-clock deterministic timed automata (DTA) specifications. The central issue is to compute the maximum probability of the set of timed paths of a CTMDP C that are accepted by a DTA A. We show that this problem can be reduced to a linear programming problem whose coefficients are maximum timed reachability probabilities in a set of CTMDPs, which are obtained via a graph decomposition of the product of the CTMDP C and the region graph of the DTA A.

[1]  Christel Baier,et al.  Model-Checking Algorithms for Continuous-Time Markov Chains , 2002, IEEE Trans. Software Eng..

[2]  Sven Schewe,et al.  Finite optimal control for time-bounded reachability in CTMDPs and continuous-time Markov games , 2010, Acta Informatica.

[3]  Mihalis Yannakakis,et al.  The complexity of probabilistic verification , 1995, JACM.

[4]  Nicolás Wolovick,et al.  A Characterization of Meaningful Schedulers for Continuous-Time Markov Decision Processes , 2006, FORMATS.

[5]  Joost-Pieter Katoen,et al.  Reachability probabilities in Markovian Timed Automata , 2011, IEEE Conference on Decision and Control and European Control Conference.

[6]  R. Bellman,et al.  Dynamic Programming and Markov Processes , 1960 .

[7]  Robert Knast.  Continuous-Time Probabilistic Automata , 1969, Inf. Control..

[8]  Peter Buchholz,et al.  Numerical analysis of continuous time Markov decision processes over finite horizons , 2011, Comput. Oper. Res..

[9]  Joost-Pieter Katoen,et al.  Quantitative Model Checking of Continuous-Time Markov Chains Against Timed Automata Specifications , 2009, 2009 24th Annual IEEE Symposium on Logic In Computer Science.

[10]  Jirí Srba,et al.  Comparing the Expressiveness of Timed Automata and Timed Extensions of Petri Nets , 2008, FORMATS.

[11]  Christel Baier,et al.  Model checking for a probabilistic branching time logic with fairness , 1998, Distributed Computing.

[12]  Rajeev Alur,et al.  A Theory of Timed Automata , 1994, Theor. Comput. Sci..

[13]  Andrea Bianco,et al.  Model Checking of Probabalistic and Nondeterministic Systems , 1995, FSTTCS.

[14]  Jan Kretínský,et al.  Stochastic Real-Time Games with Qualitative Timed Automata Objectives , 2010, CONCUR.

[15]  Paul Gastin,et al.  CONCUR 2010 - Concurrency Theory, 21th International Conference, CONCUR 2010, Paris, France, August 31-September 3, 2010. Proceedings , 2010, CONCUR.

[16]  Pawel Sobocinski,et al.  Being Van Kampen is a universal property , 2011, Log. Methods Comput. Sci..

[17]  Luca de Alfaro,et al.  How to Specify and Verify the Long-Run Average Behavior of Probabilistic Systems , 1998, LICS.

[18]  Vincent Danos,et al.  Reversible Communicating Systems , 2004, CONCUR.

[19]  Jan Kretínský,et al.  Continuous-Time Stochastic Games with Time-Bounded Reachability , 2009, FSTTCS.

[20]  Moshe Y. Vardi.  Automatic verification of probabilistic concurrent finite state programs , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[21]  Kim Guldstrand Larsen,et al.  The power of reachability testing for timed automata , 2003, Theor. Comput. Sci..

[22]  T. Henzinger,et al.  Trading memory for randomness , 2004 .

[23]  Mark H. Davis.  Markov Models and Optimization , 1995 .

[24]  Anders Martin-Löf,et al.  Optimal Control of a Continuous-Time Markov Chain with Periodic Transition Probabilities , 1967, Oper. Res..

[25]  B. L. Miller.  Finite State Continuous Time Markov Decision Processes with a Finite Planning Horizon , 1968 .

[26]  Christel Baier,et al.  Efficient computation of time-bounded reachability probabilities in uniform continuous-time Markov decision processes , 2005, Theor. Comput. Sci..

[27]  B. L. Miller.  Finite state continuous time Markov decision processes with an infinite planning horizon , 1968 .

[28]  Joost-Pieter Katoen,et al.  The Ins and Outs of the Probabilistic Model Checker MRMC , 2009, 2009 Sixth International Conference on the Quantitative Evaluation of Systems.

[29]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[30]  Christel Baier,et al.  LiQuor: A tool for Qualitative and Quantitative Linear Time analysis of Reactive Systems , 2006, Third International Conference on the Quantitative Evaluation of Systems - (QEST'06).

[31]  Serge Haddad,et al.  Model Checking Timed and Stochastic Properties with CSL^{TA} , 2009, IEEE Transactions on Software Engineering.

[32]  U. Rieder,et al.  Markov Decision Processes , 2010 .

[33]  Xi-Ren Cao,et al.  A survey of recent results on continuous-time Markov decision processes , 2006 .

[34]  Philippe Schnoebelen,et al.  Model Checking Timed Automata with One or Two Clocks , 2004, CONCUR.

[35]  Rajeev Alur,et al.  A Temporal Logic of Nested Calls and Returns , 2004, TACAS.

[36]  Jan Kretínský,et al.  Measuring performance of continuous-time stochastic processes using timed automata , 2011, HSCC '11.

[37]  Ian Stark,et al.  Free-Algebra Models for the pi-Calculus , 2005, FoSSaCS.

[38]  Lijun Zhang,et al.  Time-Bounded Reachability Probabilities in Continuous-Time Markov Decision Processes , 2010, 2010 Seventh International Conference on the Quantitative Evaluation of Systems.

[39]  Christel Baier,et al.  Model Checking Markov Chains with Actions and State Labels , 2007, IEEE Transactions on Software Engineering.

[40]  Joost-Pieter Katoen,et al.  Efficient CTMC Model Checking of Linear Real-Time Objectives , 2011, TACAS.

[41]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[42]  Joost-Pieter Katoen,et al.  Delayed Nondeterminism in Continuous-Time Markov Decision Processes , 2009, FoSSaCS.