Combining Model-Based $Q$ -Learning With Structural Knowledge Transfer for Robot Skill Learning

Learning skills autonomously is a particularly important ability for an autonomous robot. A promising approach is reinforcement learning (RL) where agents learn policy through interaction with its environment. One problem of RL algorithm is how to tradeoff the exploration and exploitation. Moreover, multiple tasks also make a great challenge to robot learning. In this paper, to enhance the performance of RL, a novel learning framework integrating RL with knowledge transfer is proposed. Three basic components are included: 1) probability policy reuse; 2) dynamic model learning; and 3) model-based <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-learning. In this framework, the prelearned skills are leveraged for policy reuse and dynamic learning. In model-based <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-learning, the Gaussian process regression is used to approximate the <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-value function so as to suit for robot control. The prior knowledge retrieved from knowledge transfer is integrated into the model-based <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-learning to reduce the needed learning time. Finally, a human-robot handover experiment is performed to evaluate the learning performance of this learning framework. Experiment results show that fewer exploration is needed to obtain a high expected reward, due to the prior knowledge obtained from knowledge transfer.