Efficient Non-Linear Control by Combining Q-learning with Local Linear Controllers

This paper presents a new approach to reinforcement learning (RL) to solve a non-linear control problem eÆciently in which state and action spaces are continuous. In real-world applications, an approach combining discrete RL methods with linear controllers is promising since there are many non-linear control problems that can be decomposed into several local linear-control tasks. We provide a hierarchical RL algorithm composed of local linear controllers and Q-learning, which are both very simple. The continuous stateaction space is discretized into an array of coarse boxes, and each box has its own local linear controller as an abstract action. The higher-level of the hierarchy is a conventional discrete RL algorithm that chooses the abstract actions. Each linear controller improves the local control policy by using an actor-critic method. The coarse state-space discretization is a quite simple way to cope with the curse of dimensionality, but often gives rise to non-Markovian e ects. In our approach, the local linear controllers make up for these undesirable e ects. The algorithm was applied to a simulation of a cartpole swing-up problem, and feasible solutions are found in less time than those of conventional discrete RL methods.