Markov chain approximation techniques for a class of nonlinear control problems

This work develops Markov chain approximation techniques for a class of nonlinear control problems in which the dynamics are given by ordinary differential equations involving a finite-state Markov chain. Our motivation comes from production planning and controls of various manufacturing systems with unreliable machines. An algorithm for the optimal control problem is developed. Our main effort is to prove the desired convergence properties of the method for approximating the optimal control and the value function via Markov chain approximation techniques. It is shown that the sequence of approximating Markov chain converges to that of the system under consideration and that the sequence of approximating value functions converges to the true value function.