Reinforcement learning to compensate for perceptual aliasing using dynamic additional parameter: motivational value

AbstnactIn this paper, we present a new reinforcement learning approach compensating for the perceptual aliasing problem by varying policies depending on the behavior context. For this approach, motiuatwd vdue(M-value) is introduced as a parameter emphasizing specific future action selection probabilities temporarily according to the context. In the learning phase, a Q-value renewal error linked with the current state-action pair is memorized as M-value linked with past visited experiences. In the control phase, t o motivate a next action, an agent awakes M-values linked with the current state and memorized in past experiences. By combining Mvalue with Q-value, even if an agent observes the same sensory inputs under the different states ,the agent can generate different action selection policies with the context. The advantage of the proposed approach is that the learning/control system reflecting the difference of context can be realized easily, in spite of the saving of computational memories, by the simple extension of general reinforcement learning: Q-learning. In order to investigate the validity of the proposed method, we apply the method to the maze problem containing perceptual aliasing problem, and compare it with the case of general Q-learning. The result on maze environment experiment shows that the proposed approach can work effectively in the non-Markov decision process environment involving perceptual aliasing problems. Keywordsreinforcement learning, Q-learning, POMDPs, perceptual aliasing.