Optimizing walking controllers for uncertain inputs and environments

We introduce methods for optimizing physics-based walking controllers for robustness to uncertainty. Many unknown factors, such as external forces, control torques, and user control inputs, cannot be known in advance and must be treated as uncertain. These variables are represented with probability distributions, and a return function scores the desirability of a single motion. Controller optimization entails maximizing the expected value of the return, which is computed by Monte Carlo methods. We demonstrate examples with different sources of uncertainty and task constraints. Optimizing control strategies under uncertainty increases robustness and produces natural variations in style.

[1]  Michiel van de Panne,et al.  Sensor-actuator networks , 1993, SIGGRAPH.

[2]  Karl Sims,et al.  Evolving virtual creatures , 1994, SIGGRAPH.

[3]  Michiel van de Panne,et al.  Guided Optimization for Balanced Locomotion , 1995 .

[4]  Demetri Terzopoulos,et al.  Automated learning of muscle-actuated locomotion through control abstraction , 1995, SIGGRAPH.

[5]  Daniel M. Wolpert,et al.  Making smooth moves , 2022 .

[6]  Geoffrey E. Hinton,et al.  NeuroAnimator: fast neural network emulation and control of physics-based models , 1998, SIGGRAPH.

[7]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[8]  Michael I. Jordan,et al.  PEGASUS: A policy search method for large MDPs and POMDPs , 2000, UAI.

[9]  Petros Faloutsos,et al.  Composable controllers for physics-based character animation , 2001, SIGGRAPH.

[10]  James C. Spall,et al.  Introduction to stochastic search and optimization - estimation, simulation, and control , 2003, Wiley-Interscience series in discrete mathematics and optimization.

[11]  Kelvin E. Jones,et al.  The scaling of motor noise with muscle strength and motor unit number in humans , 2004, Experimental Brain Research.

[12]  C. N Bouza,et al.  Spall, J.C. Introduction to stochastic search and optimization. Estimation, simulation and control. Wiley Interscience Series in Discrete Mathematics and Optimization, 2003 , 2004 .

[13]  H. Sebastian Seung,et al.  Stochastic policy gradient reinforcement learning on a simple 3D biped , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[14]  E. Todorov.  Optimality principles in sensorimotor control , 2004, Nature Neuroscience.

[15]  Jehee Lee,et al.  Precomputing avatar behavior from human motion data , 2004, SCA '04.

[16]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[17]  Michiel van de Panne,et al.  Synthesis of Controllers for Stylized Planar Bipedal Walking , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[18]  Pieter Abbeel,et al.  An Application of Reinforcement Learning to Aerobatic Helicopter Flight , 2006, NIPS.

[19]  Nikolaus Hansen,et al.  The CMA Evolution Strategy: A Comparing Review , 2006, Towards a New Evolutionary Computation.

[20]  Adrien Treuille,et al.  Near-optimal character animation with continuous control , 2007, SIGGRAPH 2007.

[21]  Jehee Lee,et al.  Simulating biped behaviors from human motion data , 2007, SIGGRAPH 2007.

[22]  Konrad P. Kording,et al.  Decision Theory: What "Should" the Nervous System Do? , 2007 .

[23]  James C. Spall,et al.  Introduction to Stochastic Search and Optimization. Estimation, Simulation, and Control (Spall, J.C. , 2007 .

[24]  Nancy S. Pollard,et al.  Responsive characters from motion fragments , 2007, SIGGRAPH 2007.

[25]  M. V. D. Panne,et al.  SIMBICON: simple biped locomotion control , 2007, SIGGRAPH 2007.

[26]  A. Faisal,et al.  Noise in the nervous system , 2008, Nature Reviews Neuroscience.

[27]  Marco da Silva,et al.  Interactive simulation of stylized human locomotion , 2008, ACM Trans. Graph..

[28]  Matthias Zwicker,et al.  Real-time planning for parameterized human motion , 2008, SCA '08.

[29]  David J. Fleet,et al.  Optimizing walking controllers , 2009, SIGGRAPH 2009.

[30]  Katie Byl,et al.  Metastable Walking Machines , 2009, Int. J. Robotics Res..

[31]  Philippe Beaudoin,et al.  Robust task-based control policies for physics-based characters , 2009, SIGGRAPH 2009.

[32]  Hans-Peter Seidel,et al.  A Statistical Model of Human Pose and Body Shape , 2009, Comput. Graph. Forum.

[33]  Zoran Popović,et al.  Contact-aware nonlinear control of dynamic characters , 2009, SIGGRAPH 2009.

[34]  Zoran Popović,et al.  Compact character controllers , 2009, SIGGRAPH 2009.