Time Discretisation and Rate of Convergence for the Optimal Control of Continuous-Time Stochastic Systems with Delay

Abstract We study a semi-discretisation scheme for stochastic optimal control problems whose dynamics are given by controlled stochastic delay (or functional) differential equations with bounded memory. Performance is measured in terms of expected costs. By discretising time in two steps, we construct a sequence of approximating finite-dimensional Markovian optimal control problems in discrete time. The corresponding value functions converge to the value function of the original problem, and we derive an upper bound on the discretisation error or, equivalently, a worst-case estimate for the rate of convergence.

[1]  Bernt Øksendal,et al.  Some Solvable Stochastic Control Problems With Delay , 2000 .

[2]  Ioannis Karatzas,et al.  Brownian Motion and Stochastic Calculus , 1987 .

[3]  Markus Fischer,et al.  Discretisation of stochastic control problems for continuous time dynamics with delay , 2006 .

[4]  Marizio Falcone,et al.  Discrete time high-order schemes for viscosity solutions of Hamilton-Jacobi-Bellman equations , 1994 .

[5]  Evelyn Buckwar,et al.  Introduction to the numerical analysis of stochastic delay differential equations , 2000 .

[6]  K. Atkinson,et al.  Theoretical Numerical Analysis: A Functional Analysis Framework , 2001 .

[7]  Harald Bauer,et al.  Stochastic control problems with delay , 2005, Math. Methods Oper. Res..

[8]  X. Zhou,et al.  Stochastic Controls: Hamiltonian Systems and HJB Equations , 1999 .

[9]  Adriano M. Garsia,et al.  A Real Variable Lemma and the Continuity of Paths of Some Gaussian Processes , 1970 .

[10]  Harish G.A Babu,et al.  Operations Research and its Application , 2007 .

[11]  Guy Barles,et al.  Error bounds for monotone approximation schemes for parabolic Hamilton-Jacobi-Bellman equations , 2007, Math. Comput..

[12]  N. Krylov.  Controlled Diffusion Processes , 1980 .

[13]  Xuerong Mao,et al.  Numerical solutions of stochastic functional differential equations , 2003 .

[14]  D. W. Stroock,et al.  Multidimensional Diffusion Processes , 1979 .

[15]  N. Krylov,et al.  Approximating Value Functions for Controlled Degenerate Diffusion Processes by Using Piece-Wise Constant Policies , 1999 .

[16]  B. Øksendal,et al.  A maximum principle for optimal control of stochastic systems with delay, with applications to finance. , 2000 .

[17]  Patrick Florchinger,et al.  Convergence in Nonlinear Filtering for Stochastic Delay Systems , 2007, SIAM J. Control. Optim..

[18]  Guy Barles,et al.  Error Bounds for Monotone Approximation Schemes for Hamilton-Jacobi-Bellman Equations , 2005, SIAM J. Numer. Anal..

[19]  M. Chang,et al.  Optimal control of stochastic functional differential equations with a bounded memory , 2008 .

[20]  M. G. Delgado,et al.  Optimal control and partial differential equations , 2004 .

[21]  S. Mohammed.  Stochastic functional differential equations , 1984 .

[22]  H. Kushner.  Numerical approximations for stochastic systems with delays in the state and control , 2006 .

[23]  N. Krylov.  On the rate of convergence of finite-difference approximations for Bellmans equations with variable coefficients , 2000 .

[24]  L. Slominski.  Euler's approximations of solutions of SDEs with reflecting boundary , 2001 .

[25]  N. Risebro,et al.  WHEN ARE HJB-EQUATIONS FOR CONTROL PROBLEMS WITH STOCHASTIC DELAY EQUATIONS FINITE DIMENSIONAL? , 2001 .

[26]  B. Larssen.  Dynamic programming in stochastic control of systems with delay , 2002 .

[27]  H. Kushner.  Numerical approximations for nonlinear stochastic systems with delays , 2005 .

[28]  L. C. G. Rogers,et al.  Pathwise Stochastic Optimal Control , 2007, SIAM J. Control. Optim..

[29]  I. Gihman,et al.  Controlled Stochastic Processes , 1979 .

[30]  N. Krylov.  Mean value theorems for stochastic integrals , 2001 .

[31]  Nils Henrik Risebro,et al.  When Are HJB-Equations in Stochastic Control of Delay Systems Finite Dimensional? , 2003 .

[32]  Salah-Eldin A. Mohammed,et al.  Discrete-time approximations of stochastic delay equations: The Milstein scheme , 2004 .

[33]  J. Quadrat.  Numerical methods for stochastic control problems in continuous time , 1994 .

[34]  K. Brown,et al.  Graduate Texts in Mathematics , 1982 .