Scaling Up Average Reward Reinforcement Learning by Approximating the Domain Models and the Value Function

Almost all the work in Average-reward Reinforcement Learning (ARL) so far has fo-cused on table-based methods which do not scale to domains with large state spaces. In this paper, we propose two extensions to a model-based ARL method called H-learning to address the scale-up problem. We extend H-learning to learn action models and reward functions in the form of Bayesian networks, and approximate its value function using local linear regression. We test our algorithms on several scheduling tasks for a simulated Automatic Guided Vehicle (AGV) and show that they are eeective in signiicantly reducing the space requirement of H-learning and making it converge faster. To the best of our knowledge, our results are the rst in applying function approximation to ARL.