On Model-Free Reinforcement Learning of Reduced-Order Optimal Control for Singularly Perturbed Systems

We propose a model-free reduced-order optimal control design for linear time-invariant singularly perturbed (SP) systems using reinforcement learning (RL). Both the state and input matrices of the plant model are assumed to be completely unknown. The only assumption imposed is that the model admits a similarity transformation that results in a SP representation. We propose a variant of Adaptive Dynamic Programming (ADP) that employs only the slow states of this SP model to learn a reduced-order adaptive optimal controller. The method significantly reduces the learning time, and complexity required for the feedback control by taking advantage of this model reduction. We use approximation theorems from singular perturbation theory to establish sub-optimality of the learned controller, and to guarantee closed-loop stability. We validate our results using two representative examples - one with a standard singularly perturbed dynamics, and the other with clustered multi-agent consensus dynamics. Both examples highlight various implementation details and effectiveness of the proposed approach.