Discretization and Weak Convergence in Markov Decision Drift Processes

In this paper we deal with continuous time Markov decision drift processes CTMDP, which permit both controls affecting jump rates of the process and impulsive controls causing immediate transitions. Between two successive jump epochs the state of the process evolves according to a deterministic drift function. Given a CTMDP we construct a sequence of discrete time Markov decision drift processes DTMDP with decreasing distance between the successive decision epochs. Sufficient conditions are provided under which the law of the CTMDP controlled by a fixed policy is the limit in the sense of weak convergence of probability measures of the laws of the approximating DTMDF's controlled by fixed discrete time policies. The conditions concern both the parameters of the CTMDP and the relation between the discrete time and continuous time policies. An application to a maintenance replacement model is given.

[1]  B. L. Miller.  Finite state continuous time Markov decision processes with an infinite planning horizon , 1968 .

[2]  R. Bellman.  Dynamic programming. , 1957, Science.

[3]  A. A. Yushkevich,et al.  Controlled Markov Models with Countable State Space and Continuous Time , 1978 .

[4]  Bharat T. Doshi,et al.  Continuous time control of Markov processes on an arbitrary state space: Average return criterion , 1976 .

[5]  K. Parthasarathy,et al.  Probability measures on metric spaces , 1967 .

[6]  P. Kakumanu,et al.  Continuous time Markovian decision processes average return criterion , 1975 .

[7]  J. Neveu,et al.  Mathematical foundations of the calculus of probability , 1965 .

[8]  Arie Hordijk,et al.  Discretization procedures for continuous time decision processes , 1979 .

[9]  Hans-Joachim Langen,et al.  Convergence of Dynamic Programming Models , 1981, Math. Oper. Res..

[10]  W. Whitt.  Weak Convergence of Probability Measures on the Function Space $C\lbrack 0, \infty)$ , 1970 .

[11]  Charles Stone.  Weak convergence of stochastic processes defined on semi-infinite time intervals , 1963 .

[12]  P. Kakumanu.  Continuously Discounted Markov Decision Model with Countable State and Action Space , 1971 .

[13]  Yu. V. Prokhorov.  Convergence of Random Processes and Limit Theorems in Probability Theory , 1956 .

[14]  F. A. van der Duyn Schouten.  Markov decision processes with continuous time parameter , 1983 .

[15]  E. Fainberg,et al.  On Homogeneous Markov Models with Continuous Time and Finite or Countable State Space , 1979 .

[16]  Ronald A. Howard,et al.  Dynamic Programming and Markov Processes , 1960 .

[17]  B. L. Miller.  Finite State Continuous Time Markov Decision Processes with a Finite Planning Horizon , 1968 .

[18]  Arie Hordijk,et al.  Weak convergence of decision processes , 1980 .

[19]  A. Skorokhod.  Limit Theorems for Stochastic Processes , 1956 .

[20]  Ward Whitt,et al.  Some Useful Functions for Functional Limit Theorems , 1980, Math. Oper. Res..

[21]  Stanley R. Pliska,et al.  Controlled jump processes , 1975 .

[22]  T. Lindvall.  Weak convergence of probability measures and random functions in the function space D [0,∞) , 1973 .

[23]  M. Robin,et al.  Contrôle impulsionnel des processus de Markov , 1978 .

[24]  P. Billingsley,et al.  Convergence of Probability Measures , 1969 .

[25]  Erhan Çinlar,et al.  A stochastic integral in storage theory , 1971 .