Time-discretization for controlled Markov processes. II. A jump and diffusion application

In a first Part I ([24]) a method of time-discretization was investigated in order to approximate continuous-time stochastic control problems over a finite time horizon. This approximation was based on using recursive discrete-time dynamic programming. To this end, three conditions are to be fulfilled: • Smoothness of the continuous-time functions • Consistency or convergence of the discrete-time generators • Stability or uniform boundedness of the discrete-time constructions. In this Part II, these conditions will be verified for two practical applications: • A contiolled infinite seivei queue, as example of a controlled Markov jump process • A contiolled cash-balance model, as example of a controlled diffusion model. For both applications it is shown and illustrated that the discrete-time constructions lead to a computational feasible scheme to approximate the optimal cost function as well as to construct an e-optimal control.

[1]  Stanley Roy Pliska.  Single and multi-person controlled diffusions , 1972 .

[2]  S. Pliska.  Single-person controlled diffusions with discounted costs , 1973 .

[3]  M. Puterman.  Sensitive Discount Optimality in Controlled One-Dimensional Diffusions , 1974 .

[4]  W. Fleming,et al.  Deterministic and Stochastic Optimal Control , 1975 .

[5]  Stanley R. Pliska,et al.  Controlled jump processes , 1975 .

[6]  George M. Constantinides,et al.  Stochastic Cash Management with Fixed and Proportional Transaction Costs , 1976 .

[7]  George M. Constantinides,et al.  Existence of Optimal Simple Policies for Discounted-Cost Inventory and Cash Management in Continuous Time , 1978, Oper. Res..

[8]  B. Doshi.  Optimal control of the service rate in an M/G/1 queueing system , 1978, Advances in Applied Probability.

[9]  I. Gihman,et al.  Controlled Stochastic Processes , 1979 .

[10]  A. A. Yushkevich.  Controlled Jump Markov Models , 1981 .

[11]  A. Bensoussan,et al.  On the Convergence of the Discrete Time Dynamic Programming Equation for General Semigroups , 1982 .

[12]  J. Michael Harrison,et al.  Instantaneous Control of Brownian Motion , 1983, Math. Oper. Res..

[13]  A. Hordijk,et al.  Average optimal policies in Markov decision drift processes with applications to a queueing and a replacement model , 1983, Advances in Applied Probability.

[14]  Norbert Christopeit.  Discrete Approximation of Continuous Time Stochastic Control Systems , 1983 .

[15]  Arie Hordijk,et al.  Discretization and Weak Convergence in Markov Decision Drift Processes , 1984, Math. Oper. Res..

[16]  N. Dijk.  Controlled Markov processes : time discretization , 1984 .

[17]  Nico M. van Dijk.  On the finite horizon Bellman equation for controlled Markov jump models with unbounded characteristics: existence and approximation , 1988 .

[18]  Arie Hordijk,et al.  Time-discretization for controlled Markov processes. I. General approximation results , 1996, Kybernetika (Praha).