Delayed Nondeterminism in Continuous-Time Markov Decision Processes

Schedulers in randomly timed games can be classified as to whether they use timing information or not. We consider continuous-time Markov decision processes (CTMDPs) and define a hierarchy of positional (P) and history-dependent (H) schedulers which induce strictly tighter bounds on quantitative properties on CTMDPs. This classification into time abstract (TA), total time (TT) and fully time-dependent (T) schedulers is mainly based on the kind of timing details that the schedulers may exploit. We investigate when the resolution of nondeterminism may be deferred. In particular, we show that TTP and TAP schedulers allow for delaying nondeterminism for all measures, whereas this does neither hold for TP nor for any TAH scheduler. The core of our study is a transformation on CTMDPs which unifies the speed of outgoing transitions per state.

[1]  Leon G. Higley,et al.  Forensic Entomology: An Introduction , 2009 .

[2]  Jirí Srba,et al.  Comparing the Expressiveness of Timed Automata and Timed Extensions of Petri Nets , 2008, FORMATS.

[3]  Christel Baier,et al.  Efficient computation of time-bounded reachability probabilities in uniform continuous-time Markov decision processes , 2005, Theor. Comput. Sci..

[4]  Robert Knast Continuous-Time Probabilistic Automata , 1969, Inf. Control..

[5]  Joost-Pieter Katoen,et al.  Bisimulation and Logical Preservation for Continuous-Time Markov Decision Processes , 2007, CONCUR.

[6]  Donald Gross,et al.  The Randomization Technique as a Modeling Tool and Solution Procedure for Transient Markov Processes , 1984, Oper. Res..

[7]  Robert B. Ash,et al.  Probability & Measure Theory , 1999 .

[8]  R. Ash,et al.  Probability and measure theory , 1999 .

[9]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[10]  A. Jensen,et al.  Markoff chains as an aid in the study of Markoff processes , 1953 .

[11]  Nicolás Wolovick,et al.  A Characterization of Meaningful Schedulers for Continuous-Time Markov Decision Processes , 2006, FORMATS.

[12]  Kim G. Larsen,et al.  On Modal Refinement and Consistency , 2007, CONCUR.

[13]  Xi-Ren Cao,et al.  A survey of recent results on continuous-time Markov decision processes , 2006 .

[14]  Christel Baier,et al.  Model-Checking Algorithms for Continuous-Time Markov Chains , 2002, IEEE Trans. Software Eng..