Analysis of Reward Functions in Deep Reinforcement Learning for Continuous State Space Control

Deep Reinforcement Learning (DRL), which uses deep neural networks for the approximation of the value function and the policy, in continuous state-space control tasks has recently shown promising results. However, the use of deep neural networks as function approximators has often resulted in intractable analyses of DRL algorithms mainly due to their non-convexities and thus a lack of theoretical guarantee such as asymptotic global convergence of the learning algorithm. Considering the fact that the reward function in reinforcement learning is one of the key entities that determines the overall characteristics of the learning agents, we focused on a smaller but an important aspect of the analysis, investigating the structure of widely used reward functions in DRL tasks and their possible effects on the learning algorithm. The proposed analysis may facilitate identification of appropriate reward functions in DRL tasks, which has often been conducted via trial and error.