Numerical approximations for stochastic systems with delays in the state and control

The Markov chain approximation numerical methods are widely used to compute optimal value functions and controls for stochastic as well as deterministic systems. We extend them to controlled general nonlinear delayed reflected diffusion models. The path, control and reflection terms can all be delayed. Previous work developed numerical approximations and convergence theorems. But when the control and reflection terms are delayed those and all other current algorithms normally lead to impossible demands on memory. An alternative “dual” approach was proposed by Kwong and Vintner for the linear deterministic system with a quadratic cost function. We extend the approach to the general nonlinear stochastic system, develop the Markov chain approximations and numerical algorithms and prove the convergence theorems. The approach reduces the memory requirement significantly. For the no-delay case, the method covers virtually all models of current interest. The method is robust and the approximations have physical interpretations as control problems closely related to the original one. These advantages carry over to the delay problem.