Maximal Cost-Bounded Reachability Probability on Continuous-Time Markov Decision Processes

In this paper, we consider multi-dimensional maximal cost-bounded reachability probability over continuous-time Markov decision processes (CTMDPs). Our major contributions are as follows. Firstly, we derive an integral characterization which states that the maximal cost-bounded reachability probability function is the least fixed-point of a system of integral equations. Secondly, we prove that the maximal cost-bounded reachability probability can be attained by a measurable deterministic cost-positional scheduler. Thirdly, we provide a numerical approximation algorithm for maximal cost-bounded reachability probability. We present these results under the setting of both early and late schedulers. Besides, we correct a fundamental proof error in the PhD Thesis by Martin Neuhauser on maximal time-bounded reachability probability by completely new proofs for the more general case of multi-dimensional maximal cost-bounded reachability probability.

[1]  Joost-Pieter Katoen,et al.  Delayed Nondeterminism in Continuous-Time Markov Decision Processes , 2009, FoSSaCS.

[2]  Holger Hermanns,et al.  Improving time bounded reachability computations in interactive Markov chains , 2015, Sci. Comput. Program..

[3]  Alexander Schrijver,et al.  Theory of linear and integer programming , 1986, Wiley-Interscience series in discrete mathematics and optimization.

[4]  Martin R. Neuhäußer,et al.  Model checking nondeterministic and randomly timed systems , 2010 .

[5]  Sven Schewe,et al.  Finite optimal control for time-bounded reachability in CTMDPs and continuous-time Markov games , 2010, Acta Informatica.

[6]  Peter Buchholz,et al.  Numerical analysis of continuous time Markov decision processes over finite horizons , 2011, Comput. Oper. Res..

[7]  Nicolás Wolovick,et al.  A Characterization of Meaningful Schedulers for Continuous-Time Markov Decision Processes , 2006, FORMATS.

[8]  Christel Baier,et al.  Reachability in continuous-time Markov reward decision processes , 2008, Logic and Automata.

[9]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1967 .

[10]  Lijun Zhang,et al.  Efficient approximation of optimal control for continuous-time Markov games , 2016, Inf. Comput..

[11]  Feller William,et al.  An Introduction To Probability Theory And Its Applications , 1950 .

[12]  Onésimo Hernández-Lerma,et al.  Selected Topics on Continuous-Time Controlled Markov Chains and Markov Games , 2012 .

[13]  Lijun Zhang,et al.  Time-Bounded Reachability Probabilities in Continuous-Time Markov Decision Processes , 2010, 2010 Seventh International Conference on the Quantitative Evaluation of Systems.

[14]  Jan Kretínský,et al.  Continuous-Time Stochastic Games with Time-Bounded Reachability , 2013, FSTTCS.

[15]  Christel Baier,et al.  Performance evaluation and model checking join forces , 2010, Commun. ACM.

[16]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .