Approximation in Model-Based Learning

Model-based reinforcement learning, in which a model of the environment's dynamics is learned and used to supplement direct learning from experience, has been proposed as a general approach to learning and planning. We present experiments with this idea in which the model of the environment's dynamics is both approximate and learned on-line. These experiments involve the Mountain Car task, which requires approximation of both value function and model because it has continuous state variables. Naive model use is susceptible to modelling errors and can impair the value function. We show that excessive model use performs worse than using no model at all. Hybrid methods can mitigate learning with inherently incorrect models .