On Decidability of Time-bounded Reachability in CTMDPs

We consider the time-bounded reachability problem for continuous-time Markov decision processes. We show that the problem is decidable subject to Schanuel's conjecture. Our decision procedure relies on the structure of optimal policies and the conditional decidability (under Schanuel's conjecture) of the theory of reals extended with exponential and trigonometric functions over bounded domains. We further show that any unconditional decidability result would imply unconditional decidability of the bounded continuous Skolem problem, or equivalently, the problem of checking if an exponential polynomial has a non-tangential zero in a bounded interval. We note that the latter problems are also decidable subject to Schanuel's conjecture but finding unconditional decision procedures remain longstanding open problems.

[1]  Christel Baier,et al.  Efficient Computation of Time-Bounded Reachability Probabilities in Uniform Continuous-Time Markov Decision Processes , 2005, TACAS.

[2]  Lijun Zhang,et al.  Model Checking Algorithms for CTMDPs , 2011, CAV.

[3]  Jean-Charles Delvenne,et al.  The continuous Skolem-Pisot problem , 2010, Theor. Comput. Sci..

[4]  Martin R. Neuhäußer,et al.  Model checking nondeterministic and randomly timed systems , 2010 .

[5]  Christel Baier,et al.  On Skolem-hardness and saturation points in Markov decision processes , 2020, ICALP.

[6]  Rupak Majumdar,et al.  A Lyapunov Approach for Time-Bounded Reachability of CTMCs and CTMDPs , 2020, ACM Trans. Model. Perform. Evaluation Comput. Syst..

[7]  James Worrell,et al.  When is Containment Decidable for Probabilistic Automata? , 2018, ICALP.

[8]  Nicolás Wolovick,et al.  A Characterization of Meaningful Schedulers for Continuous-Time Markov Decision Processes , 2006, FORMATS.

[9]  Joël Ouaknine,et al.  Reachability problems for Markov chains , 2015, Inf. Process. Lett..

[10]  Angus Macintyre,et al.  On the decidability of the real exponential field , 1996 .

[11]  Lijun Zhang,et al.  Efficient approximation of optimal control for continuous-time Markov games , 2016, Inf. Comput..

[12]  A. Wilkie.  Schanuel’s Conjecture and the Decidability of the Real Exponential Field , 1997 .

[13]  Christel Baier,et al.  Principles of model checking , 2008 .

[14]  Joost-Pieter Katoen,et al.  Delayed Nondeterminism in Continuous-Time Markov Decision Processes , 2009, FoSSaCS.

[15]  Angus Macintyre.  Turing meets Schanuel , 2016, Ann. Pure Appl. Log..

[16]  Jan Kretínský,et al.  Continuous-Time Stochastic Games with Time-Bounded Reachability , 2013, FSTTCS.

[17]  Peter Buchholz,et al.  Numerical analysis of continuous time Markov decision processes over finite horizons , 2011, Comput. Oper. Res..

[18]  Calin Belta,et al.  Formal Synthesis of Control Policies for Continuous Time Markov Processes From Time-Bounded Temporal Logic Specifications , 2014, IEEE Transactions on Automatic Control.

[19]  Angus Macintyre.  Model theory of exponentials on Lie algebras , 2008, Math. Struct. Comput. Sci..

[20]  B. L. Miller.  Finite State Continuous Time Markov Decision Processes with a Finite Planning Horizon , 1968 .

[21]  Lijun Zhang,et al.  Time-Bounded Reachability Probabilities in Continuous-Time Markov Decision Processes , 2010, 2010 Seventh International Conference on the Quantitative Evaluation of Systems.

[22]  Joël Ouaknine,et al.  On the Skolem Problem for Continuous Linear Dynamical Systems , 2015, ICALP.

[23]  Sven Schewe,et al.  Finite optimal control for time-bounded reachability in CTMDPs and continuous-time Markov games , 2010, Acta Informatica.

[24]  Sven Schewe,et al.  Optimal time-abstract schedulers for CTMDPs and continuous-time Markov games , 2013, Theor. Comput. Sci..