On Determinism Handling While Learning Reduced State Space Representations

When applying a Reinforcement Learning technique to problems with continuous or very large state spaces, some kind of generalization is required. In the bibliography, two main approaches can be found. On one hand, the generalization problem can be defined as an approximation problem of the continuous value function, typically solved with neural networks. On the other hand, other approaches discretize or cluster the states of the original state space to achieve a reduced one in order to learn a discrete value table. However, both methods have disadvantages, like the introduction of non-determinism in the discretizations, parameters hard to tune by the user, or the use of a high number of resources. In this paper, we use some characteristics of both approaches to achieve state space representations that allow to approximate the value function in deterministic reinforcement learning problems. The method clusters the domain supervised by the value function being learned to avoid the non-determinism introduction. At the same time, the size of the new representation stays small and it is automatically computed. Experiments show improvements over other approaches such as uniform or unsupervised clustering.