Continuous time markov decision processes with interventions

We study Markov jump decision processes with both continuously and instantaneouslyacting decisions and with deterministic drift between jumps. Such decision processes were recentlyintroduced and studied from discrete time approximations point of view by Van der Duyn Schouten.Weobtain necessary and sufficient optimality conditions for these decision processes in terms of equations and inequalities of quasi-variational type. By means of the latter we find simple necessaryand sufficient conditions for the existence of stationary optimal policies in such processes with finite state and action spaces, both in the discounted and average per unit time reward cases.

[1]  P. Kakumanu.  Continuously Discounted Markov Decision Model with Countable State and Action Space , 1971 .

[2]  John G. Kemeny,et al.  Finite Markov chains , 1960 .

[3]  E. Fainberg,et al.  On Homogeneous Markov Models with Continuous Time and Finite or Countable State Space , 1979 .

[4]  Martin L. Puterman,et al.  Finite Generalized Markov Programming. , 1982 .

[5]  Bharat T. Doshi,et al.  Continuous time control of Markov processes on an arbitrary state space: Average return criterion , 1976 .

[6]  Awi Federgruen,et al.  A GENERAL MARKOV DECISION METHOD I: MODEL AND TECHNIQUES , 1977 .

[7]  Bharat T. Doshi,et al.  Generalized semi-Markov decision processes , 1979, Journal of Applied Probability.

[8]  P. Kakumanu,et al.  Nondiscounted Continuous Time Markovian Decision Process with Countable State Space , 1972 .

[9]  B. L. Miller.  Finite state continuous time Markov decision processes with an infinite planning horizon , 1968 .

[10]  Jose Luis Menaldi,et al.  On the Optimal Impulse Control Problem for Degenerate Diffusions , 1980 .

[11]  D. Blackwell.  Discrete Dynamic Programming , 1962 .

[12]  P. Kakumanu,et al.  Continuous time Markovian decision processes average return criterion , 1975 .

[13]  B. L. Miller.  Finite State Continuous Time Markov Decision Processes with a Finite Planning Horizon , 1968 .

[14]  Stanley R. Pliska,et al.  Controlled jump processes , 1975 .

[15]  R. Bellman.  Dynamic programming. , 1957, Science.

[16]  Onésimo Hernández-Lerma,et al.  Controlled Markov Processes , 1965 .

[17]  N. Krylov.  Controlled Diffusion Processes , 1980 .

[18]  Alain Bensoussan,et al.  Applications des Inequations Varia-tionnelles en Controle Stochastique , 1978 .

[19]  Ronald A. Howard,et al.  Dynamic Programming and Markov Processes , 1960 .

[20]  E. Denardo,et al.  Multichain Markov Renewal Programs , 1968 .

[21]  J. Neveu.  Bases mathématiques du calcul des probabilités , 1966 .