Constrained Continuous-Time Markov Control Processes with Discounted Criteria

In this paper we study constrained continuous-time Markov control processes with a denumerable state space and unbounded reward/cost and transition rates. The criterion to be maximized is an expected discounted reward, and the constraint is imposed on an expected discounted cost. We give conditions that ensure the existence of constrained-optimal policies. We also show that a constrained-optimal policy may be a stationary policy or a randomized stationary policy that randomizes between two stationary policies which differ in at most one state. Our results are illustrated with a controlled queueing system.

[1]  M. Puterman,et al.  Bias optimality in controlled queueing systems , 1998 .

[2]  Eugene A. Feinberg,et al.  Constrained Discounted Dynamic Programming , 1996, Math. Oper. Res..

[3]  Onésimo Hernández-Lerma,et al.  Constrained Markov control processes in Borel spaces: the discounted case , 2000, Math. Methods Oper. Res..

[4]  Kensuke Tanaka.  On discounted dynamic programming with constraints , 1991 .

[5]  E. Altman.  Constrained Markov Decision Processes , 1999 .

[6]  Weiping Zhu,et al.  Denumerable state continuous time Markov decision processes with unbounded cost and transition rates under average criterion , 2002, The ANZIAM Journal.

[7]  Martin L. Puterman,et al.  A note on bias optimality in controlled queueing systems , 2000, Journal of Applied Probability.

[8]  L. Sennott.  Stochastic Dynamic Programming and the Control of Queueing Systems , 1998 .

[9]  Onésimo Hernández-Lerma,et al.  Nonstationary Continuous-Time Markov Control Processes with Discounted Costs on Infinite Horizon , 2001 .

[10]  Kai Lai Chung,et al.  Markov Chains with Stationary Transition Probabilities , 1961 .

[11]  S. Meyn,et al.  Stability of Markovian processes III: Foster–Lyapunov criteria for continuous-time processes , 1993, Advances in Applied Probability.

[12]  A. Piunovskiy.  Optimal Control of Random Sequences in Problems with Constraints , 1997 .

[13]  Xianping Guo,et al.  Optimality Conditions for CTMDP with Average Cost Criterion , 2002 .

[14]  Xianping Guo,et al.  Continuous-Time Controlled Markov Chains with Discounted Rewards , 2003 .

[15]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[16]  F. Beutler,et al.  Optimal policies for controlled markov chains with a constraint , 1985 .

[17]  Xuerong Mao,et al.  Constrained Markovian decision processes: the dynamic programming approach , 2000, Oper. Res. Lett..

[18]  Raquiel R López Martínez,et al.  The Lagrange and Pareto approaches to constrained Markov control processes , 2000 .

[19]  Masami Kurano,et al.  Constrained markov decision processes with compact state and action spaces: the average case , 2000 .

[20]  O. Hernández-Lerma,et al.  Further topics on discrete-time Markov control processes , 1999 .

[21]  Xianping Guo,et al.  A note on optimality conditions for continuous-time Markov decision processes with average cost criterion , 2001, IEEE Trans. Autom. Control..

[22]  John S. Edwards,et al.  Linear Programming and Finite Markovian Control Problems , 1983 .

[23]  E. Fainberg,et al.  On Homogeneous Markov Models with Continuous Time and Finite or Countable State Space , 1979 .

[24]  Arie Hordijk,et al.  Constrained Undiscounted Stochastic Dynamic Programming , 1984, Math. Oper. Res..

[25]  Linn I. Sennott,et al.  Constrained Discounted Markov Decision Chains , 1991, Probability in the Engineering and Informational Sciences.