The Effects of Intrinsic Motivation Signals on Reinforcement Learning Strategies

Using neurobiological and psychological models in robotics and machine learning was of growing interest in the last years. Whereas common algorithms in the reinforcement learning framework tend to get stuck in local maxima while exploring the environment, intrinsic motivation modules can be used to extend these algorithms and push the reinforcement learning agent out of its equilibrium, similar to a human who gets bored of a task he fulfills many times or makes no progress while trying to fulfill it. This thesis gives an overview of models of intrinsic motivation founded on neurobiology and psychology, before presenting a computational view of extending algorithms of the reinforcement learning framework with intrinsic motivation models. Several existing theoretical models and related work are presented, achieving a better performance than classic algorithms, regarding the exploration/exploitation trade-off and driving the autonomous learning of an agent. Three of these models, maximizing incompetence motivation (IM), maximizing competence motivation (CM) and competence progress motivation (CPM), are implemented, in which the authors define competence by the number of primitive actions an agent needs to reach a terminal state and add a negative intrinsic reward for reaching this terminal state, which increases or decreases proportionally to the competence of the agent. The models are evaluated on four simulated scenarios and compared with the performance of the classic reinforcement learning algorithm SARSA and a time-decreasing-epsilon (TDE) modification of it. Using CM achieves at best a faster convergence towards the same terminal state as SARSA, whereas using IM and CPM results in an agent being pushed out of its equilibrium of local maxima and leads to more exploration, while still maximizing the expected external reward. An agent using these models is able to learn skills which an agent using classic SARSA would never explore. The presented related work and the implemented models show that using models for intrinsic motivation together with reinforcement learning algorithms results in well-performing behavior for tasks, on which classic algorithms would fail or get stuck in local maxima, and so provide a useful base for future work and research to build up a fully autonomous learning system.