Dual Online Stein Variational Inference for Control and Dynamics

Model predictive control (MPC) schemes have a proven track record for delivering aggressive and robust performance in many challenging control tasks, coping with nonlinear system dynamics, constraints, and observational noise. Despite their success, these methods often rely on simple control distributions, which can limit their performance in highly uncertain and complex environments. MPC frameworks must be able to accommodate changing distributions over system parameters, based on the most recent measurements. In this paper, we devise an implicit variational inference algorithm able to estimate distributions over model parameters and control inputs on-the-fly. The method incorporates Stein Variational gradient descent to approximate the target distributions as a collection of particles, and performs updates based on a Bayesian formulation. This enables the approximation of complex multi-modal posterior distributions, typically occurring in challenging and realistic robot navigation tasks. We demonstrate our approach on both simulated and realworld experiments requiring real-time execution in the face of dynamically changing environments.

[1]  Takuya Akiba,et al.  Optuna: A Next-generation Hyperparameter Optimization Framework , 2019, KDD.

[2]  E. Todorov,et al.  A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems , 2005, Proceedings of the 2005, American Control Conference, 2005..

[3]  Peter Jan van Leeuwen,et al.  Sequential Monte Carlo with kernel embedded mappings: The mapping particle filter , 2019, J. Comput. Phys..

[4]  James M. Rehg,et al.  Robust Sampling Based Model Predictive Control with Sparse Objective Information , 2018, Robotics: Science and Systems.

[5]  Byron Boots,et al.  Prediction under Uncertainty in Sparse Spectrum Gaussian Processes with Applications to Filtering and Control , 2017, ICML.

[6]  D. Fox,et al.  Stein Variational Model Predictive Control , 2020, CoRL.

[7]  Evangelos A. Theodorou,et al.  Model Predictive Path Integral Control: From Theory to Parallel Computation , 2017 .

[8]  Dieter Fox,et al.  BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators , 2019, Robotics: Science and Systems.

[9]  Daoliang Li,et al.  Model predictive control and its application in agriculture: A review , 2018, Comput. Electron. Agric..

[10]  Carl E. Rasmussen,et al.  PILCO: A Model-Based and Data-Efficient Approach to Policy Search , 2011, ICML.

[11]  Dariusz Pazderski,et al.  Modeling and control of a 4-wheel skid-steering mobile robot , 2004 .

[12]  D. Fox,et al.  Online BayesSim for Combined Simulator Parameter Inference and Policy Improvement , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[13]  Dezhen Song,et al.  Kinematic Modeling and Analysis of Skid-Steered Mobile Robots With Applications to Low-Cost Inertial-Measurement-Unit-Based Motion Estimation , 2009, IEEE Transactions on Robotics.

[14]  Dirk P. Kroese,et al.  Chapter 3 – The Cross-Entropy Method for Optimization , 2013 .

[15]  Byron Boots,et al.  Adaptive Probabilistic Trajectory Optimization via Efficient Approximate Inference , 2016, ArXiv.

[16]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[17]  Roland Potthast,et al.  Particle filters for applications in geosciences , 2018, 1807.10434.

[18]  Marc Toussaint,et al.  On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference , 2012, Robotics: Science and Systems.

[19]  Marco Pavone,et al.  Meta-Learning Priors for Efficient Online Bayesian Regression , 2018, WAFR.

[20]  Ali-akbar Agha-mohammadi,et al.  Deep Learning Tubes for Tube MPC , 2020, Robotics: Science and Systems.

[21]  Dilin Wang,et al.  Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm , 2016, NIPS.

[22]  Dirk P. Kroese,et al.  Kernel density estimation via diffusion , 2010, 1011.2602.

[23]  James M. Rehg,et al.  Information-Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving , 2017, IEEE Transactions on Robotics.

[24]  M Maarten Steinbuch,et al.  Optimization and optimal control in automotive systems , 2014 .

[25]  Kim P. Wabersich,et al.  Bayesian model predictive control: Efficient model exploration and regret bounds using posterior sampling , 2020, L4DC.

[26]  Roland Siegwart,et al.  Model Predictive Control for Trajectory Tracking of Unmanned Aerial Vehicles Using Robot Operating System , 2017 .

[27]  Tadahiro Taniguchi,et al.  Variational Inference MPC for Bayesian Model-based Reinforcement Learning , 2019, CoRL.

[28]  Sergey Levine,et al.  Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models , 2018, NeurIPS.

[29]  Lionel Ott,et al.  DISCO: Double Likelihood-free Inference Stochastic Control , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[30]  John W. Eaton,et al.  Model Predictive Control of Chemical Processes , 1991, 1991 American Control Conference.

[31]  Stefano Di Cairano,et al.  Automotive Applications of Model Predictive Control , 2018, Handbook of Model Predictive Control.

[32]  Yuval Tassa,et al.  Control-limited differential dynamic programming , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[33]  Jaime F. Fisac,et al.  A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems , 2017, IEEE Transactions on Automatic Control.

[34]  Edward Johns,et al.  Crossing The Gap: A Deep Dive into Zero-Shot Sim-to-Real Transfer for Dynamics , 2020, ArXiv.

[35]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[36]  Nolan Wagener,et al.  An Online Learning Approach to Model Predictive Control , 2019, Robotics: Science and Systems.

[37]  Yunpeng Pan,et al.  Probabilistic Differential Dynamic Programming , 2014, NIPS.

[38]  Wolfram Burgard,et al.  Improved Techniques for Grid Mapping With Rao-Blackwellized Particle Filters , 2007, IEEE Transactions on Robotics.

[39]  Timothy J. Robinson,et al.  Sequential Monte Carlo Methods in Practice , 2003 .

[40]  Jan Peters,et al.  Stochastic Optimal Control as Approximate Input Inference , 2019, CoRL.

[41]  Dieter Fox,et al.  Model-Based Generalization Under Parameter Uncertainty Using Path Integral Control , 2020, IEEE Robotics and Automation Letters.