Efficient computation of time-bounded reachability probabilities in uniform continuous-time Markov decision processes

A continuous-time Markov decision process (CTMDP) is a generalization of a continuous-time Markov chain in which both probabilistic and nondeterministic choices co-exist. This paper presents an efficient algorithm to compute the maximum (or minimum) probability to reach a set of goal states within a given time bound in a uniform CTMDP, i.e., a CTMDP in which the delay time distribution per state visit is the same for all states. It furthermore proves that these probabilities coincide for (time-abstract) history-dependent and Markovian schedulers that resolve nondeterminism either deterministically or in a randomized way.

[1]  William H. Sanders,et al.  An efficient well-specified check , 1999, Proceedings 8th International Workshop on Petri Nets and Performance Models (Cat. No.PR00331).

[2]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[3]  P. Buchholz.  Exact and ordinary lumpability in finite Markov chains , 1994, Journal of Applied Probability.

[4]  Andrea Bianco,et al.  Model Checking of Probabalistic and Nondeterministic Systems , 1995, FSTTCS.

[5]  Holger Hermanns,et al.  A tool for model-checking Markov chains , 2003, International Journal on Software Tools for Technology Transfer.

[6]  John G. Kemeny,et al.  Finite Markov Chains. , 1960 .

[7]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[8]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1967 .

[9]  Holger Hermanns,et al.  Stochastic Process Algebras - Between LOTOS and Markov Chains , 1998, Comput. Networks.

[10]  Luca de Alfaro,et al.  How to Specify and Verify the Long-Run Average Behavior of Probabilistic Systems , 1998, LICS.

[11]  Peter W. Glynn,et al.  Computing Poisson probabilities , 1988, CACM.

[12]  Christel Baier,et al.  Model-Checking Algorithms for Continuous-Time Markov Chains , 2002, IEEE Trans. Software Eng..

[13]  Marco Ajmone Marsan,et al.  Modelling with Generalized Stochastic Petri Nets , 1995, PERV.

[14]  John G. Kemeny,et al.  Finite Markov chains , 1960 .

[15]  Giovanni Chiola,et al.  GSPNs versus SPNs: what is the actual role of immediate transitions? , 1991, Proceedings of the Fourth International Workshop on Petri Nets and Performance Models PNPM91.

[16]  Christel Baier,et al.  An efficient algorithm tocompute the minimal and maximal probabilities for timed reachability incontinuous-time Markov decision processes , 2003 .

[17]  Marta Z. Kwiatkowska,et al.  Probabilistic symbolic model checking with PRISM: a hybrid approach , 2004, International Journal on Software Tools for Technology Transfer.

[18]  V. Kulkarni.  Modeling and Analysis of Stochastic Systems , 1996 .

[19]  Joost-Pieter Katoen,et al.  Model-checking large structured Markov chains , 2003, J. Log. Algebraic Methods Program..

[20]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1951 .

[21]  Keith W. Ross,et al.  Uniformization for semi-Markov decision processes under stationary policies , 1987, Journal of Applied Probability.

[22]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[23]  Christel Baier,et al.  Comparative Branching-Time Semantics , 2003, CONCUR.

[24]  Joost-Pieter Katoen,et al.  Faster and Symbolic CTMC Model Checking , 2001, PAPM-PROBMIV.

[25]  L. Sennott.  Stochastic Dynamic Programming and the Control of Queueing Systems , 1998 .

[26]  Kim G. Larsen,et al.  Reduction and Refinement Strategies for Probabilistic Analysis , 2002, PAPM-PROBMIV.

[27]  William H. Sanders,et al.  Reduced Base Model Construction Methods for Stochastic Activity Networks , 1991, IEEE J. Sel. Areas Commun..

[28]  Zohar Manna,et al.  Formal verification of probabilistic systems , 1997 .

[29]  Eugene Asarin,et al.  On Optimal Scheduling under Uncertainty , 2003, TACAS.

[30]  Robert K. Brayton,et al.  Model-checking continuous-time Markov chains , 2000, TOCL.

[31]  Christel Baier,et al.  Model checking for a probabilistic branching time logic with fairness , 1998, Distributed Computing.

[32]  Eugene A. Feinberg,et al.  Continuous Time Discounted Jump Markov Decision Processes: A Discrete-Event Approach , 2004, Math. Oper. Res..

[33]  Greg N. Frederickson,et al.  Sequencing Tasks with Exponential Service Times to Minimize the Expected Flow Time or Makespan , 1981, JACM.

[34]  Gianfranco Ciardo,et al.  Well-defined stochastic Petri nets , 1996, Proceedings of MASCOTS '96 - 4th International Workshop on Modeling, Analysis and Simulation of Computer and Telecommunication Systems.

[35]  Holger Hermanns,et al.  Interactive Markov Chains , 2002, Lecture Notes in Computer Science.

[36]  Massoud Pedram,et al.  Dynamic power management based on continuous-time Markov decision processes , 1999, DAC '99.