We are interested in the problem of determining a course of action to achieve a desired objective in a non-deterministic environment. Markov decision processes (MDPs) provide a framework for representing this action selection problem, and there are a number of algorithms that learn optimal policies within this formulation. This framework has also been used to study state space abstraction, problem decomposition, and policy reuse. These techniques sacrifice optimality of their solution for improved learning speed. In this paper we examine the suboptimality of reusing policies that are solutions to subproblems. This is done within a restricted class of MDPs, namely those where non-zero reward is received only upon reaching a goal state. We introduce the definition of a subproblem within this class and provide motivation for how reuse of subproblem solutions can speed up learning. The contribution of this paper is the derivation of a tight bound on the loss in optimality from this reuse. We examine a bound that is based on Bellman error, which applies to all MDPs, but is not tight enough to be useful. We contribute our own theoretical result that gives an empirically tight bound on this suboptimality.
[1]
Manuela M. Veloso,et al.
Flexible Strategy Learning: Analogical Replay of Problem Solving Episodes
,
1994,
AAAI.
[2]
Ronald J. Williams,et al.
Tight Performance Bounds on Greedy Policies Based on Imperfect Value Functions
,
1993
.
[3]
Milos Hauskrecht,et al.
Hierarchical Solution of Markov Decision Processes using Macro-actions
,
1998,
UAI.
[4]
Doina Precup,et al.
Between MOPs and Semi-MOP: Learning, Planning & Representing Knowledge at Multiple Temporal Scales
,
1998
.
[5]
Sebastian Thrun,et al.
Finding Structure in Reinforcement Learning
,
1994,
NIPS.
[6]
Verzekeren Naar Sparen,et al.
Cambridge
,
1969,
Humphrey Burton: In My Own Time.
[7]
Ronald Parr,et al.
Flexible Decomposition Algorithms for Weakly Coupled Markov Decision Problems
,
1998,
UAI.
[8]
Thomas Dean,et al.
Decomposition Techniques for Planning in Stochastic Domains
,
1995,
IJCAI.
[9]
Andrew W. Moore,et al.
Reinforcement Learning: A Survey
,
1996,
J. Artif. Intell. Res..