Catch & Carry

We address the longstanding challenge of producing flexible, realistic humanoid character controllers that can perform diverse whole-body tasks involving object interactions. This challenge is central to a variety of fields, from graphics and animation to robotics and motor neuroscience. Our physics-based environment uses realistic actuation and first-person perception - including touch sensors and egocentric vision - with a view to producing active-sensing behaviors (e.g. gaze direction), transferability to real robots, and comparisons to the biology. We develop an integrated neural-network based approach consisting of a motor primitive module, human demonstrations, and an instructed reinforcement learning regime with curricula and task variations. We demonstrate the utility of our approach for several tasks, including goal-conditioned box carrying and ball catching, and we characterize its behavioral robustness. The resulting controllers can be deployed in real-time on a standard PC.1

[1]  Demis Hassabis,et al.  A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play , 2018, Science.

[2]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[3]  NohJunyong,et al.  Model Predictive Control with a Visuomotor System for Physics-based Character Animation , 2020 .

[4]  Glen Berseth,et al.  DeepLoco: dynamic locomotion skills using hierarchical deep reinforcement learning , 2017, ACM Trans. Graph..

[5]  Sergey Levine,et al.  Space-time planning with parameterized locomotion controllers , 2011, TOGS.

[6]  Scott L. Delp,et al.  A Model of the Upper Extremity for Simulating Musculoskeletal Surgery and Analyzing Neuromuscular Control , 2005, Annals of Biomedical Engineering.

[7]  Okan Arikan,et al.  Interactive motion generation from examples , 2002, ACM Trans. Graph..

[8]  Leslie Pack Kaelbling,et al.  Effective reinforcement learning for mobile robots , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[9]  Zoran Popovic,et al.  Discovery of complex behaviors through contact-invariant optimization , 2012, ACM Trans. Graph..

[10]  Pieter Abbeel,et al.  Learning for control from multiple demonstrations , 2008, ICML '08.

[11]  Dana H. Ballard,et al.  Modeling embodied visual behaviors , 2007, TAP.

[12]  Yuval Tassa,et al.  Learning and Transfer of Modulated Locomotor Controllers , 2016, ArXiv.

[13]  Stefan Jeschke,et al.  Physics-based motion capture imitation with deep reinforcement learning , 2018, MIG.

[14]  Stefan Schaal,et al.  Learning from Demonstration , 1996, NIPS.

[15]  Kyoungmin Lee,et al.  Scalable muscle-actuated human simulation and control , 2019, ACM Trans. Graph..

[16]  Aude Billard,et al.  Catching Objects in Flight , 2014, IEEE Transactions on Robotics.

[17]  Jan Peters,et al.  Policy Search for Motor Primitives in Robotics , 2008, NIPS 2008.

[18]  Michiel van de Panne,et al.  Sensor-actuator networks , 1993, SIGGRAPH.

[19]  Nando de Freitas,et al.  Reinforcement and Imitation Learning for Diverse Visuomotor Skills , 2018, Robotics: Science and Systems.

[20]  Kazuya Otani,et al.  Adaptive whole-body manipulation in human-to-humanoid multi-contact motion retargeting , 2017, 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids).

[21]  Peter Englert,et al.  Learning manipulation skills from a single demonstration , 2018, Int. J. Robotics Res..

[22]  Jan Peters,et al.  Learning motor primitives for robotics , 2009, 2009 IEEE International Conference on Robotics and Automation.

[23]  Oussama Khatib,et al.  Synthesis of Whole-Body Behaviors through Hierarchical Control of Behavioral Primitives , 2005, Int. J. Humanoid Robotics.

[24]  Lucas Kovar,et al.  Motion graphs , 2002, SIGGRAPH Classes.

[25]  Scott Niekum,et al.  Learning and generalization of complex tasks from unstructured demonstrations , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[26]  Demetri Terzopoulos,et al.  Animat vision: Active vision in artificial animals , 1995, Proceedings of IEEE International Conference on Computer Vision.

[27]  Stefan Schaal,et al.  Computational approaches to motor learning by imitation. , 2003, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[28]  Joonho Lee,et al.  Learning agile and dynamic motor skills for legged robots , 2019, Science Robotics.

[29]  Bilge Mutlu,et al.  Authoring directed gaze for full-body motion capture , 2016, ACM Trans. Graph..

[30]  Libin Liu,et al.  Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning , 2018, ACM Trans. Graph..

[31]  Greg Wayne,et al.  Hierarchical motor control in mammals and machines , 2019, Nature Communications.

[32]  Sebastian Starke,et al.  Neural state machine for character-scene interactions , 2019, ACM Trans. Graph..

[33]  Aude Billard,et al.  Reinforcement learning for imitating constrained reaching movements , 2007, Adv. Robotics.

[34]  Stefan Schaal,et al.  2008 Special Issue: Reinforcement learning of motor skills with policy gradients , 2008 .

[35]  Rolf Pfeifer,et al.  Understanding intelligence , 2020, Inequality by Design.

[36]  Wojciech Zaremba,et al.  Domain randomization for transferring deep neural networks from simulation to the real world , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[37]  Maja J. Mataric,et al.  Automated derivation of behavior vocabularies for autonomous humanoid motion , 2003, AAMAS '03.

[38]  Jessica K. Hodgins,et al.  Animation of dynamic legged locomotion , 1991, SIGGRAPH.

[39]  Jessica K. Hodgins,et al.  Interactive control of avatars animated with human motion data , 2002, SIGGRAPH.

[40]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Stefan Schaal,et al.  Skill learning and task outcome prediction for manipulation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[42]  Sergey Levine,et al.  DeepMimic , 2018, ACM Trans. Graph..

[43]  Alberto Rodriguez,et al.  TossingBot: Learning to Throw Arbitrary Objects With Residual Physics , 2019, IEEE Transactions on Robotics.

[44]  Sergey Levine,et al.  (CAD)$^2$RL: Real Single-Image Flight without a Single Real Image , 2016, Robotics: Science and Systems.

[45]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[46]  Tao Zhou,et al.  Deep learning of biomimetic sensorimotor control for biomechanical human animation , 2018, ACM Trans. Graph..

[47]  Baining Guo,et al.  Terrain runner , 2012, ACM Trans. Graph..

[48]  Yuval Tassa,et al.  MuJoCo: A physics engine for model-based control , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[49]  Daniel Holden,et al.  DReCon: data-driven responsive control of physics-based characters , 2019, ACM Trans. Graph..

[50]  Sunmin Lee,et al.  Learning predict-and-simulate policies from unorganized human motion data , 2019, ACM Trans. Graph..

[51]  A. Frank van der Stappen,et al.  Data-driven Gaze Animation using Recurrent Neural Networks , 2019, MIG.

[52]  Glen Berseth,et al.  Terrain-adaptive locomotion skills using deep reinforcement learning , 2016, ACM Trans. Graph..

[53]  Dinesh K. Pai,et al.  Eyecatch: simulating visuomotor coordination for object interception , 2012, ACM Trans. Graph..