Switching control in multi-mode Markov decision processes

This paper presents a switching control strategy for multi-mode Markov decision processes. The system to be controlled is modeled as a finite-state controlled Markov chain with a mode that evolves stochastically. Although the system state is observable, the mode is only partially observable in the sense that we know the system mode only when it is in a given set of observable modes. Given a set of controllers for the system, we consider the problem of determining a switching rule that selects the controller to be applied each time the system mode is observable. The objective is to minimize a long-term average cost from the system while satisfying bounds on the long-term average of other given performance measures. We assume the multi-mode model parameters are unknown a priori, so an adaptive switching rule is required. Algorithms are presented for computing approximations to the optimal switching rule based on estimating the model parameters online. The approach is illustrated for an example of dynamic power management of hard disk drives in computer systems.