Using Predictive Representations to Improve Generalization in Reinforcement Learning

The predictive representations hypothesis holds that particularly good generalization will result from representing the state of the world in terms of predictions about possible future experience. This hypothesis has been a central motivation behind recent research in, for example, PSRs and TD networks. In this paper we present the first explicit investigation of this hypothesis. We show in a reinforcement-learning example (a grid-world navigation task) that a predictive representation in tabular form can learn much faster than both the tabular explicit-state representation and a tabular history-based method.

[1]  Ronald L. Rivest,et al.  Diversity-based inference of finite automata , 1994, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[2]  Mark B. Ring.  Continual learning in reinforcement environments , 1995, GMD-Bericht.

[3]  Andrew McCallum,et al.  Reinforcement learning with selective perception and hidden state , 1996 .

[4]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[5]  Richard S. Sutton,et al.  Predictive Representations of State , 2001, NIPS.

[6]  Doina Precup,et al.  A Planning Algorithm for Predictive State Representations , 2003, IJCAI.

[7]  Matthew W. Mitchell,et al.  Using Markov-k Memory for Problems with Hidden-State , 2003, MLMTA.

[8]  Michael L. Littman,et al.  Planning with predictive state representations , 2004, 2004 International Conference on Machine Learning and Applications, 2004. Proceedings..

[9]  Richard S. Sutton,et al.  Temporal-Difference Networks , 2004, NIPS.

[10]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.