Optimal Control of Markov Processes with Age-Dependent Transition Rates

We study optimal control of Markov processes with age-dependent transition rates. The control policy is chosen continuously over time based on the state of the process and its age. We study infinite horizon discounted cost and infinite horizon average cost problems. Our approach is via the construction of an equivalent semi-Markov decision process. We characterise the value function and optimal controls for both discounted and average cost cases.

[1]  A. Hordijk,et al.  Average optimal policies in Markov decision drift processes with applications to a queueing and a replacement model , 1983, Advances in Applied Probability.

[2]  L. Rogers.  Stochastic differential equations and diffusion processes: Nobuyuki Ikeda and Shinzo Watanabe North-Holland, Amsterdam, 1981, xiv + 464 pages, Dfl.175.00 , 1982 .

[3]  Stanley R. Pliska,et al.  Controlled jump processes , 1975 .

[4]  A. Federgruen,et al.  Denumerable state semi-markov decision processes with unbounded costs, average cost criterion : (preprint) , 1979 .

[5]  O. Hernández-Lerma,et al.  Continuous-time controlled Markov chains , 2003 .

[6]  S. Ethier,et al.  Markov Processes: Characterization and Convergence , 2005 .

[7]  Arie Hordijk,et al.  Markov Decision Drift Processes; Conditions for Optimality Obtained by Discretization , 1985, Math. Oper. Res..

[8]  Mrinal K. Ghosh,et al.  Risk Minimizing Option Pricing in a Semi-Markov Modulated Market , 2009, SIAM J. Control. Optim..

[9]  Xi-Ren Cao,et al.  A survey of recent results on continuous-time Markov decision processes , 2006 .

[10]  Frank A. Van der Duyn Schouten.  Markov decision drift processes , 1986 .

[11]  D. Aldous.  Review: Stewart N. Ethier and Thomas G. Kurtz, Markov processes: Characterization and convergence , 1987 .

[12]  Xianping Guo,et al.  Continuous-Time Markov Decision Processes: Theory and Applications , 2009 .

[13]  Arie Hordijk,et al.  Discretization and Weak Convergence in Markov Decision Drift Processes , 1984, Math. Oper. Res..

[14]  S. Lippman.  Semi-Markov Decision Processes with Unbounded Rewards , 1973 .

[15]  Paul J. Schweitzer,et al.  Denumerable Undiscounted Semi-Markov Decision Processes with Unbounded Rewards , 1983, Math. Oper. Res..

[16]  Mark H. Davis.  Markov Models and Optimization , 1995 .

[17]  K. Wakuta,et al.  Arbitrary state semi-Markov decision processes , 1987 .

[18]  池田 信行,et al.  Stochastic differential equations and diffusion processes , 1981 .