Adaptive Path-Integral Approach to Representation Learning and Planning for Dynamical Systems

We present a representation learning algorithm that learns a low-dimensional latent dynamical system from high-dimensional \textit{sequential} raw data, e.g., video. The framework builds upon recent advances in amortized inference methods that use both an inference network and a refinement procedure to output samples from a variational distribution given an observation sequence, and takes advantage of the duality between control and inference to approximately solve the intractable inference problem using the path integral control approach. The learned dynamical model can be used to predict and plan the future states; we also present the efficient planning method that exploits the learned low-dimensional latent dynamics. Numerical experiments show that the proposed path-integral control based variational inference method leads to tighter lower bounds in statistical model learning of sequential data. The supplementary video: https://youtu.be/xCp35crUoLQ

[1]  David J. Fleet,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE Gaussian Process Dynamical Model , 2007 .

[2]  David Duvenaud,et al.  Reinterpreting Importance-Weighted Autoencoders , 2017, ICLR.

[3]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[4]  Yee Whye Teh,et al.  Filtering Variational Objectives , 2017, NIPS.

[5]  Martin A. Riedmiller,et al.  Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images , 2015, NIPS.

[6]  David Duvenaud,et al.  Inference Suboptimality in Variational Autoencoders , 2018, ICML.

[7]  David Filliat,et al.  State Representation Learning for Control: An Overview , 2018, Neural Networks.

[8]  Scott W. Linderman,et al.  Variational Sequential Monte Carlo , 2017, AISTATS.

[9]  Hilbert J. Kappen,et al.  Particle Smoothing for Hidden Diffusion Processes: Adaptive Path Integral Smoother , 2016, IEEE Transactions on Signal Processing.

[10]  Emanuel Todorov,et al.  Efficient computation of optimal actions , 2009, Proceedings of the National Academy of Sciences.

[11]  Ole Winther,et al.  A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning , 2017, NIPS.

[12]  Maximilian Karl,et al.  Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data , 2016, ICLR.

[13]  Emanuel Todorov,et al.  General duality between optimal control and estimation , 2008, 2008 47th IEEE Conference on Decision and Control.

[14]  Ryan P. Adams,et al.  Composing graphical models with neural networks for structured representations and fast inference , 2016, NIPS.

[15]  Oliver Brock,et al.  Learning state representations with robotic priors , 2015, Auton. Robots.

[16]  Marco Pavone,et al.  Learning Sampling Distributions for Robot Motion Planning , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[17]  C. Gardiner.  Handbook of Stochastic Methods , 1983 .

[18]  Ali Ghodsi,et al.  Robust Locally-Linear Controllable Embedding , 2017, AISTATS.

[19]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[20]  Andriy Mnih,et al.  Variational Inference for Monte Carlo Objectives , 2016, ICML.

[21]  Daniel D. Lee,et al.  Learning Implicit Sampling Distributions for Motion Planning , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[22]  Maximilian Karl,et al.  Dynamic movement primitives in latent space of time-dependent variational autoencoders , 2016, 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids).

[23]  Tuan Anh Le,et al.  Auto-Encoding Sequential Monte Carlo , 2017, ICLR.

[24]  Daniel D. Lee,et al.  Learning and exploiting low-dimensional structure for efficient holonomic motion planning in high-dimensional spaces , 2012, Int. J. Robotics Res..

[25]  Matthew D. Hoffman,et al.  On the challenges of learning with inference networks on sparse, high-dimensional data , 2017, AISTATS.

[26]  Luca Rigazio,et al.  Path Integral Networks: End-to-End Differentiable Optimal Control , 2017, ArXiv.

[27]  Pieter Abbeel,et al.  Value Iteration Networks , 2016, NIPS.

[28]  Han-Lim Choi,et al.  Approximate Inference-Based Motion Planning by Learning and Exploiting Low-Dimensional Latent Variable Models , 2018, IEEE Robotics and Automation Letters.

[29]  David Hsu,et al.  QMDP-Net: Deep Learning for Planning under Partial Observability , 2017, NIPS.

[30]  H. Kappen,et al.  Path integral control and state-dependent feedback. , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[31]  Alexander M. Rush,et al.  Semi-Amortized Variational Autoencoders , 2018, ICML.

[32]  Nebojsa Jojic,et al.  Iterative Refinement of the Approximate Posterior for Directed Belief Networks , 2015, NIPS.

[33]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[34]  H. Kappen,et al.  Adaptive Importance Sampling for Control and Inference , 2016, Journal of Statistical Physics.

[35]  Uri Shalit,et al.  Structured Inference Networks for Nonlinear State Space Models , 2016, AAAI.