In reinforcement learning the interaction between the agent and the environment generally takes place on a fixed time scale, which means that the control interval is set to a fixed time step. In order to determine a suitable fixed time scale one has to trade off accuracy in control against learning complexity. In this paper, we present an alternative approach that enables the agent to learn a control policy by using multiple time scales simultaneously. Instead of preselecting a fixed time scale, there are several time scales available during learning and the agent can select the appropriate time scale depending on the system state. The different time scales are multiples of a finest time scale which is denoted as the primitive time scale. Actions on a coarser time scale consist of several identical actions on the primitive time scale and are called multistep actions (MSAs). The special structure of these actions is efficiently exploited in our recently proposed MSA-Q-learning algorithm. In this paper, we use the MSAs to learn a control policy for a thermostat control problem. Our algorithm yields a fast and highly accurate control policy; in contrast, the standard Q-learning algorithms without MSAs fails to learn any useful control policy for this problem.
[1]
Martin A. Riedmiller.
Concepts and Facilities of a Neural Reinforcement Learning Control Architecture for Technical Process Control
,
1999,
Neural Computing & Applications.
[2]
Richard S. Sutton,et al.
Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding
,
1995,
NIPS.
[3]
Thomas G. Dietterich.
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
,
1999,
J. Artif. Intell. Res..
[4]
Doina Precup,et al.
Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning
,
1999,
Artif. Intell..
[5]
Ronald E. Parr,et al.
Hierarchical control and learning for markov decision processes
,
1998
.
[6]
Stephan Pareigis,et al.
Adaptive Choice of Grid and Time in Reinforcement Learning
,
1997,
NIPS.
[7]
Martin A. Riedmiller,et al.
Speeding-up Reinforcement Learning with Multi-step Actions
,
2002,
ICANN.