An adjustment method of the number of states of Q-learning segmenting state space adaptively

This paper presents a method to partition a continuous state space for the purposes of realizing an autonomous behavior of agent. The basic idea of this partitioning technique is derived from QLASS (Q-learning with adaptive state segmentation) which is a simple and effective technique. In segmentation by QLASS, since discrete state space is constructed as Voronoi diagram which is generated by a set of a finite number of points called generators, the state space is intuitively easy to understand. However, as QLASS has a problem that the algorithm generates too many segments in which during the learning, an agent, which uses QLASS, cannot learn appropriate action efficiently. To overcome this problem, an adjustment method of the number of states is proposed, method which restricts or boosts the partitioning by using eligibilities and temperature parameter of each segment. Experimental results show that this adjustment method can partition state space suitably according to not only the environment characteristic but its dynamic changes.