Dynamic Programming, Q-learning and other discrete Markov Decision Process solvers can be applied to continuous d-dimensional state-spaces by quantizing the state space into an array of boxes. This is often problematic above two dimensions: a coarse quantization can lead to poor policies, and fine quantization is too expensive. Possible solutions are variable-resolution discretization, or function approximation by neural nets. A third option, which has been little studied in the reinforcement learning literature, is interpolation on a coarse grid. In this paper we study interpolation techniques that can result in vast improvements in the online behavior of the resulting control systems: multilinear interpolation, and an interpolation algorithm based on an interesting regular triangulation of d-dimensional space. We adapt these interpolators under three reinforcement learning paradigms: (i) offline value iteration with a known model, (ii) Q-learning, and (iii) online value iteration with a previously unknown model learned from data. We describe empirical results, and the resulting implications for practical learning of continuous non-linear dynamic control.
[1]
G. D. Liveing,et al.
The University of Cambridge
,
1897,
British medical journal.
[2]
D. Moore.
Simplicial Mesh Generation with Applications
,
1992
.
[3]
C. Atkeson,et al.
Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time
,
1993,
Machine Learning.
[4]
Andrew W. Moore,et al.
Generalization in Reinforcement Learning: Safely Approximating the Value Function
,
1994,
NIPS.
[5]
Geoffrey J. Gordon.
Stable Function Approximation in Dynamic Programming
,
1995,
ICML.
[6]
Ben J. A. Kröse,et al.
Learning from delayed rewards
,
1995,
Robotics Auton. Syst..
[7]
Andrew G. Barto,et al.
Improving Elevator Performance Using Reinforcement Learning
,
1995,
NIPS.
[8]
Richard S. Sutton,et al.
Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding
,
1995,
NIPS.
[9]
Thomas G. Dietterich.
What is machine learning?
,
2020,
Archives of Disease in Childhood.