Solving Large POMDPs using Real Time Dynamic Programming

Partially Observable Markov Decision Processes (pomdps) are general models of sequential decision problems in which both actions and observations may be noisy. Many problems of interest can be formulated as pomdps yet the use of pomdps has been limited by the lack of eeective algorithms: optimal algorithms don't scale up and heuristic algorithms often do poorly. In this paper, a new pomdp algorithm is introduced that combines the beneets of optimal and heuristic procedures producing good solutions quickly even in problems that are large. Like optimal procedures, the procedure rtdp-bel attempts to solve the information mdp, yet like heuristic procedures, it makes decisions in real time following a suitable heuristic function. rtdp-bel is a Real Time Dynamic Programming algorithm 1], namely, a greedy search algorithm that learns to solve mdps by repeatedly updating the heuristic values of the states that are visited. As shown by Barto et al. such updates eventually deliver an optimal behavior provided that the state space is nite and the initial heuristic values are admissible. Since information mdps have innnite state spaces we discretize probabilities and combine them with heuristic values obtained from the underlying mdp. Although the resulting algorithm is not guaranteed to be optimal, experiments over a number of benchmarks suggest that large pomdps are quickly and consistently solved, and that solutions, if not optimal, tend to be very good.