Learning State-Dependent Losses for Inverse Dynamics Learning

Being able to quickly adapt to changes in dynamics is paramount in model-based control for object manipulation tasks. In order to influence fast adaptation of the inverse dynamics model’s parameters, data efficiency is crucial. Given observed data, a key element to how an optimizer updates model parameters is the loss function. In this work, we propose to apply meta-learning to learn structured, state-dependent loss functions during a meta-training phase. We then replace standard losses with our learned losses during online adaptation tasks. We evaluate our proposed approach on inverse dynamics learning tasks, both in simulation and on real hardware data. In both settings, the structured and state-dependent learned losses improve online adaptation speed, when compared to standard, state-independent loss functions.

[1]  Christopher G. Atkeson,et al.  Estimation of Inertial Parameters of Manipulator Loads and Links , 1986 .

[2]  Stefan Schaal,et al.  Local dimensionality reduction for locally weighted learning , 1997, Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA'97. 'Towards New Computational Principles for Robotics and Automation'.

[3]  Christopher G. Atkeson,et al.  Constructive Incremental Learning from Only Local Information , 1998, Neural Computation.

[4]  Stefan Schaal,et al.  Scalable Techniques from Nonparametric Statistics for Real Time Robot Learning , 2002, Applied Intelligence.

[5]  Ricardo Vilalta,et al.  A Perspective View and Survey of Meta-Learning , 2002, Artificial Intelligence Review.

[6]  Duy Nguyen-Tuong,et al.  Local Gaussian Process Regression for Real Time Online Model Learning , 2008, NIPS.

[7]  G. Evans,et al.  Learning to Optimize , 2008 .

[8]  Jan Peters,et al.  Using model knowledge for learning inverse dynamics , 2010, 2010 IEEE International Conference on Robotics and Automation.

[9]  Olivier Sigaud,et al.  On-line regression algorithms for learning mechanical models of robots: A survey , 2011, Robotics Auton. Syst..

[10]  Jan Peters,et al.  Model learning for robot control: a survey , 2011, Cognitive Processing.

[11]  Giorgio Metta,et al.  Real-time model learning using Incremental Sparse Spectrum Gaussian Process Regression. , 2013, Neural networks : the official journal of the International Neural Network Society.

[12]  Lorenzo Jamone,et al.  Incremental learning of context-dependent dynamic internal models for robot control , 2014, 2014 IEEE International Symposium on Intelligent Control (ISIC).

[13]  Reza Shadmehr,et al.  A memory of errors in sensorimotor learning , 2014, Science.

[14]  Stefan Schaal,et al.  Incremental Local Gaussian Regression , 2014, NIPS.

[15]  Marcin Andrychowicz,et al.  Learning to learn by gradient descent by gradient descent , 2016, NIPS.

[16]  Stefan Schaal,et al.  DOOMED: Direct Online Optimization of Modeling Errors in Dynamics , 2016, Big Data.

[17]  Giorgio Metta,et al.  Incremental semiparametric inverse dynamics learning , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Stefan Schaal,et al.  Drifting Gaussian processes with varying neighborhood sizes for online model learning , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[19]  Stefan Schaal,et al.  Towards robust online inverse dynamics learning , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[20]  Sergey Levine,et al.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , 2017, ICML.

[21]  Stefan Schaal,et al.  Online Learning of a Memory for Learning Rates , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Pieter Abbeel,et al.  Evolved Policy Gradients , 2018, NeurIPS.

[23]  Sergey Levine,et al.  Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL , 2018, ICLR.

[24]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[25]  Sergey Levine,et al.  Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning , 2018, ICLR.

[26]  Stefan Schaal,et al.  Learning and Adaptation of Inverse Dynamics Models: A Comparison , 2019, 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids).

[27]  Artem Molchanov,et al.  Generalized Inner Loop Meta-Learning , 2019, ArXiv.

[28]  Austin S. Wang,et al.  Encoding Physical Constraints in Differentiable Newton-Euler Algorithm , 2020, L4DC.

[29]  Jean-Baptiste Mouret,et al.  Fast Online Adaptation in Robotics through Meta-Learning Embeddings of Simulated Priors , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[30]  Timothy M. Hospedales,et al.  Meta-Learning in Neural Networks: A Survey , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Yevgen Chebotar,et al.  Meta Learning via Learned Loss , 2019, 2020 25th International Conference on Pattern Recognition (ICPR).