Discounted Stochastic Fluid Programs

We consider optimal control problems for stochastic fluid models of the following type: Suppose ( Z t ) is a continuous-time Markov chain with finite state space. As long asZ t=z, the dynamics of the system at timet are given by a functionb z ( u(i¾·)), whereu is a control we have to choose. A cost rate functionc is given, depending on the state and the action. We want to control the system in such a way as to minimize the expected discounted cost over an infinite horizon. We will call a problem of this type aStochastic Fluid Program (SFP). They typically appear in production and telecommunication systems. We formulate the optimization problem as a discrete time Markov decision process and give conditions under which an optimal stationary policy exists. Furthermore, we show how to solve SFPs numerically, using Kushner's approximating Markov chain approach. Last but not least, we apply our results to a multiproduct manufacturing system without backlog.