Model-based Motion Imitation for Agile, Diverse and Generalizable Quadupedal Locomotion

Robots operating in human environments need a variety of skills, like slow and fast walking, turning, and sidestepping. However, building robot controllers that can exhibit such a large range of behaviors is challenging, and unsolved. We present an approach that uses a model-based controller for imitating different animal gaits without requiring any realworld fine-tuning. Unlike previous works that learn one policy per motion, we present a unified controller which is capable of generating four different animal gaits on the A1 robot. Our framework includes a trajectory optimization procedure that improves the quality of real-world imitation. We demonstrate our results in simulation and on a real 12-DoF A1 quadruped robot. Our result shows that our approach can mimic four animal motions, and outperform baselines learned per motion.

[1]  J. Grizzle,et al.  Angular Momentum about the Contact Point for Control of Bipedal Locomotion: Validation in a LIP-based Controller , 2020, ArXiv.

[2]  Daniel E. Koditschek,et al.  Spring loaded inverted pendulum running: a plant model , 1998 .

[3]  Taku Komura,et al.  Mode-adaptive neural networks for quadruped motion control , 2018, ACM Trans. Graph..

[4]  Kazuhito Yokoi,et al.  A realtime pattern generator for biped walking , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[5]  Glen Berseth,et al.  Feedback Control For Cassie With Deep Reinforcement Learning , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[6]  Christopher G. Atkeson,et al.  Using Deep Reinforcement Learning to Learn High-Level Policies on the ATRIAS Biped , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[7]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[8]  Abhinav Gupta,et al.  Neural Dynamic Policies for End-to-End Sensorimotor Learning , 2020, NeurIPS.

[9]  Stelian Coros,et al.  Animal Gaits on Quadrupedal Robots Using Motion Matching and Model-Based Control , 2021, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[10]  Tucker Hermans,et al.  Active Learning of Probabilistic Movement Primitives , 2019, 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids).

[11]  Olivier Sigaud,et al.  Path Integral Policy Improvement with Covariance Matrix Adaptation , 2012, ICML.

[12]  Oliver Kroemer,et al.  Learning to select and generalize striking movements in robot table tennis , 2012, AAAI Fall Symposium: Robots Learning Interactively from Human Teachers.

[13]  Nikolaus Hansen,et al.  The CMA Evolution Strategy: A Comparing Review , 2006, Towards a New Evolutionary Computation.

[14]  Jun Nakanishi,et al.  Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors , 2013, Neural Computation.

[15]  Xingye Da,et al.  GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model , 2021, ArXiv.

[16]  Marc H. Raibert,et al.  Legged Robots That Balance , 1986, IEEE Expert.

[17]  Kristin Musselman,et al.  Infant stepping: a window to the behaviour of the human pattern generator for walking. , 2004, Canadian journal of physiology and pharmacology.

[18]  Michiel van de Panne,et al.  ALLSTEPS: Curriculum‐driven Learning of Stepping Stone Skills , 2020, Comput. Graph. Forum.

[19]  Jungdam Won,et al.  A scalable approach to control diverse behaviors for physically simulated characters , 2020, ACM Trans. Graph..

[20]  Yee Whye Teh,et al.  Neural probabilistic motor primitives for humanoid control , 2018, ICLR.

[21]  Jan Peters,et al.  Learning motor primitives for robotics , 2009, 2009 IEEE International Conference on Robotics and Automation.

[22]  Donghyun Kim,et al.  Highly Dynamic Quadruped Locomotion via Whole-Body Impulse Control and Model Predictive Control , 2019, ArXiv.

[23]  Stefan Schaal,et al.  Skill learning and task outcome prediction for manipulation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[24]  Jungdam Won,et al.  Learning body shape variation in physics-based characters , 2019, ACM Trans. Graph..

[25]  Byron Boots,et al.  Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion , 2020, Conference on Robot Learning.

[26]  Jie Tan,et al.  Learning Agile Robotic Locomotion Skills by Imitating Animals , 2020, RSS 2020.

[27]  Sangbae Kim,et al.  Dynamic Locomotion in the MIT Cheetah 3 Through Convex Model-Predictive Control , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[28]  Sangbae Kim,et al.  MIT Cheetah 3: Design and Control of a Robust, Dynamic Quadruped Robot , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[29]  Yuandong Tian,et al.  Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion , 2020, ArXiv.

[30]  Sergey Levine,et al.  DeepMimic , 2018, ACM Trans. Graph..

[31]  Trista Pei-chun Chen,et al.  CARL , 2020, ACM Trans. Graph..

[32]  Stefan Schaal,et al.  Learning feedback terms for reactive planning and control , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[33]  Yuan F. Zheng,et al.  Pattern generation using coupled oscillators for robotic and biorobotic adaptive periodic movement , 1997, Proceedings of International Conference on Robotics and Automation.

[34]  Yuval Tassa,et al.  Learning human behaviors from motion capture by adversarial imitation , 2017, ArXiv.

[35]  Sunmin Lee,et al.  Learning predict-and-simulate policies from unorganized human motion data , 2019, ACM Trans. Graph..

[36]  Darwin G. Caldwell,et al.  Robot motor skill coordination with EM-based Reinforcement Learning , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[37]  Vítor M. F. Santos,et al.  Adaptation of Robot Locomotion Patterns with Dynamic Movement Primitives , 2015, 2015 IEEE International Conference on Autonomous Robot Systems and Competitions.

[38]  Auke Jan Ijspeert,et al.  Central pattern generators for locomotion control in animals and robots: A review , 2008, Neural Networks.

[39]  Jun Morimoto,et al.  Task-Specific Generalization of Discrete and Periodic Dynamic Movement Primitives , 2010, IEEE Transactions on Robotics.

[40]  Alan Fern,et al.  Learning Spring Mass Locomotion: Guiding Policies with a Reduced-Order Model , 2020, ArXiv.

[41]  Glen Berseth,et al.  Terrain-adaptive locomotion skills using deep reinforcement learning , 2016, ACM Trans. Graph..