Learning Predictive State Representations for planning

Predictive State Representations (PSRs) allow modeling of dynamical systems directly in observables and without relying on latent variable representations. A problem that arises from learning PSRs is that it is often hard to attribute semantic meaning to the learned representation. This makes generalization and planning in PSRs challenging. In this paper, we extend PSRs and introduce the notion of PSRs that include prior information (P-PSRs) to learn representations which are suitable for planning and interpretation. By learning a low-dimensional embedding of test features we map belief points of similar semantic to the same region of a subspace. This facilitates better generalization for planning and semantical interpretation of the learned representation. In specific, we show how to overcome the training sample bias and introduce feature selection such that the resulting representation emphasizes observables related to the planning task. We show that our P-PSRs result in qualitatively meaningful representations and present quantitative results that indicate improved suitability for planning.

[1]  Michael I. Jordan,et al.  Distance Metric Learning with Application to Clustering with Side-Information , 2002, NIPS.

[2]  Joelle Pineau,et al.  Goal-Directed Online Learning of Predictive Models , 2011, EWRL.

[3]  Martin A. Riedmiller,et al.  Autonomous reinforcement learning on raw visual input data in a real world application , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[4]  Michael R. James,et al.  Learning predictive state representations in dynamical systems without reset , 2005, ICML.

[5]  Richard S. Sutton,et al.  Predictive Representations of State , 2001, NIPS.

[6]  Michael R. James,et al.  Predictive State Representations: A New Theory for Modeling Dynamical Systems , 2004, UAI.

[7]  Byron Boots,et al.  An Online Spectral Learning Algorithm for Partially Observable Nonlinear Dynamical Systems , 2011, AAAI.

[8]  Danica Kragic,et al.  Learning Predictive State Representation for in-hand manipulation , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[9]  Martin A. Riedmiller.  Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method , 2005, ECML.

[10]  Olivier Pietquin,et al.  Subspace identification for predictive state representation by nuclear norm minimization , 2014, 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL).

[11]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[12]  Terrence J. Sejnowski,et al.  Slow Feature Analysis: Unsupervised Learning of Invariances , 2002, Neural Computation.

[13]  Joelle Pineau,et al.  Efficient learning and planning with compressed predictive states , 2013, J. Mach. Learn. Res..

[14]  Sebastian Thrun,et al.  Learning low dimensional predictive representations , 2004, ICML.

[15]  Doina Precup,et al.  Point-Based Planning for Predictive State Representations , 2008, Canadian Conference on AI.

[16]  L. P. Kaelbling,et al.  Learning Geometrically-Constrained Hidden Markov Models for Robot Navigation: Bridging the Topological-Geometrical Gap , 2011, J. Artif. Intell. Res..

[17]  Byron Boots,et al.  Closing the learning-planning loop with predictive state representations , 2009, Int. J. Robotics Res..

[18]  Nan Jiang,et al.  Spectral Learning of Predictive State Representations with Insufficient Statistics , 2015, AAAI.

[19]  Byron Boots,et al.  Hilbert Space Embeddings of Predictive State Representations , 2013, UAI.

[20]  Jan Peters,et al.  Model learning for robot control: a survey , 2011, Cognitive Processing.

[21]  Sebastian Thrun,et al.  Monte Carlo POMDPs , 1999, NIPS.

[22]  Michael H. Bowling,et al.  Action respecting embedding , 2005, ICML.

[23]  Satinder P. Singh,et al.  On discovery and learning of models with predictive representations of state for agents with continuous actions and observations , 2007, AAMAS '07.

[24]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Oliver Brock,et al.  State Representation Learning in Robotics: Using Prior Knowledge about Physical Interaction , 2014, Robotics: Science and Systems.

[26]  Pierre Geurts,et al.  Tree-Based Batch Mode Reinforcement Learning , 2005, J. Mach. Learn. Res..

[27]  Joelle Pineau,et al.  Modelling Sparse Dynamical Systems with Compressed Predictive State Representations , 2013, ICML.

[28]  Leslie Pack Kaelbling,et al.  Planning and Acting in Partially Observable Stochastic Domains , 1998, Artif. Intell..