Towards a common implementation of reinforcement learning for multiple robotic tasks

Abstract Mobile robots are increasingly being employed for performing complex tasks in dynamic environments. Those tasks can be either explicitly programmed by an engineer or learned by means of some automatic learning method, which improves the adaptability of the robot and reduces the effort of setting it up. In this sense, reinforcement learning (RL) methods are recognized as a promising tool for a machine to learn autonomously how to do tasks that are specified in a relatively simple manner. However, the dependency between these methods and the particular task to learn is a well-known problem that has strongly restricted practical implementations in robotics so far. Breaking this barrier would have a significant impact on these and other intelligent systems; in particular, having a core method that requires little tuning effort for being applicable to diverse tasks would boost their autonomy in learning and self-adaptation capabilities. In this paper we present such a practical core implementation of RL, which enables the learning process for multiple robotic tasks with minimal per-task tuning or none. Based on value iteration methods, we introduce a novel approach for action selection, called Q-biased softmax regression (QBIASSR), that takes advantage of the structure of the state space by attending the physical variables involved (e.g., distances to obstacles, robot pose, etc.), thus experienced sets of states accelerate the decision-making process of unexplored or rarely-explored states. Intensive experiments with both real and simulated robots, carried out with the software framework also introduced here, show that our implementation is able to learn different robotic tasks without tuning the learning method. They also suggest that the combination of true online SARSA(λ) (TOSL) with QBIASSR can outperform the existing RL core algorithms in low-dimensional robotic tasks. All of these are promising results towards the possibility of learning much more complex tasks autonomously by a robotic agent.