Optimizing walking controllers for uncertain inputs and environments

We introduce methods for optimizing physics-based walking controllers for robustness to uncertainty. Many unknown factors, such as external forces, control torques, and user control inputs, cannot be known in advance and must be treated as uncertain. These variables are represented with probability distributions, and a return function scores the desirability of a single motion. Controller optimization entails maximizing the expected value of the return, which is computed by Monte Carlo methods. We demonstrate examples with different sources of uncertainty and task constraints. Optimizing control strategies under uncertainty increases robustness and produces natural variations in style.

[1]  Michiel van de Panne,et al.  Sensor-actuator networks , 1993, SIGGRAPH.

[2]  Karl Sims,et al.  Evolving virtual creatures , 1994, SIGGRAPH.

[3]  Michiel van de Panne,et al.  Guided Optimization for Balanced Locomotion , 1995 .

[4]  Demetri Terzopoulos,et al.  Automated learning of muscle-actuated locomotion through control abstraction , 1995, SIGGRAPH.

[5]  Daniel M. Wolpert,et al.  Signal-dependent noise determines motor planning , 1998, Nature.

[6]  Geoffrey E. Hinton,et al.  NeuroAnimator: fast neural network emulation and control of physics-based models , 1998, SIGGRAPH.

[7]  Michael I. Jordan,et al.  PEGASUS: A policy search method for large MDPs and POMDPs , 2000, UAI.

[8]  Petros Faloutsos,et al.  Composable controllers for physics-based character animation , 2001, SIGGRAPH.

[9]  James C. Spall,et al.  Introduction to stochastic search and optimization - estimation, simulation, and control , 2003, Wiley-Interscience series in discrete mathematics and optimization.

[10]  Kelvin E. Jones,et al.  The scaling of motor noise with muscle strength and motor unit number in humans , 2004, Experimental Brain Research.

[11]  H. Sebastian Seung,et al.  Stochastic policy gradient reinforcement learning on a simple 3D biped , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[12]  E. Todorov Optimality principles in sensorimotor control , 2004, Nature Neuroscience.

[13]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.

[14]  Jehee Lee,et al.  Precomputing avatar behavior from human motion data , 2004, SCA '04.

[15]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[16]  Michiel van de Panne,et al.  Synthesis of Controllers for Stylized Planar Bipedal Walking , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[17]  Pieter Abbeel,et al.  An Application of Reinforcement Learning to Aerobatic Helicopter Flight , 2006, NIPS.

[18]  Nikolaus Hansen,et al.  The CMA Evolution Strategy: A Comparing Review , 2006, Towards a New Evolutionary Computation.

[19]  Z. Popovic,et al.  Near-optimal character animation with continuous control , 2007, ACM Trans. Graph..

[20]  Jehee Lee,et al.  Simulating biped behaviors from human motion data , 2007, SIGGRAPH 2007.

[21]  Nancy S. Pollard,et al.  Responsive characters from motion fragments , 2007, SIGGRAPH 2007.

[22]  M. V. D. Panne,et al.  SIMBICON: simple biped locomotion control , 2007, SIGGRAPH 2007.

[23]  Konrad Paul Kording,et al.  Decision Theory: What "Should" the Nervous System Do? , 2007, Science.

[24]  A. Faisal,et al.  Noise in the nervous system , 2008, Nature Reviews Neuroscience.

[25]  Marco da Silva,et al.  Interactive simulation of stylized human locomotion , 2008, ACM Trans. Graph..

[26]  Matthias Zwicker,et al.  Real-time planning for parameterized human motion , 2008, SCA '08.

[27]  Katie Byl,et al.  Metastable Walking Machines , 2009, Int. J. Robotics Res..

[28]  David J. Fleet,et al.  Optimizing walking controllers , 2009, ACM Trans. Graph..

[29]  Philippe Beaudoin,et al.  Robust task-based control policies for physics-based characters , 2009, SIGGRAPH 2009.

[30]  Hans-Peter Seidel,et al.  A Statistical Model of Human Pose and Body Shape , 2009, Comput. Graph. Forum.

[31]  Zoran Popović,et al.  Contact-aware nonlinear control of dynamic characters , 2009, SIGGRAPH 2009.

[32]  Zoran Popović,et al.  Compact character controllers , 2009, SIGGRAPH 2009.