Linear Bellman combination for control of character animation

Controllers are necessary for physically-based synthesis of character animation. However, creating controllers requires either manual tuning or expensive computer optimization. We introduce linear Bellman combination as a method for reusing existing controllers. Given a set of controllers for related tasks, this combination creates a controller that performs a new task. It naturally weights the contribution of each component controller by its relevance to the current state and goal of the system. We demonstrate that linear Bellman combination outperforms naive combination often succeeding where naive combination fails. Furthermore, this combination is provably optimal for a new task if the component controllers are also optimal for related tasks. We demonstrate the applicability of linear Bellman combination to interactive character control of stepping motions and acrobatic maneuvers.

[1]  Zoran Popovic,et al.  Physically based motion transformation , 1999, SIGGRAPH.

[2]  C. K. Liu,et al.  Learning physics-based motion style with nonlinear inverse optimization , 2005, SIGGRAPH 2005.

[3]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[4]  Emanuel Todorov,et al.  Efficient computation of optimal actions , 2009, Proceedings of the National Academy of Sciences.

[5]  Philippe Beaudoin,et al.  Synthesis of constrained walking skills , 2008, SIGGRAPH 2008.

[6]  Philippe Beaudoin,et al.  Continuation methods for adapting simulated skills , 2008, SIGGRAPH 2008.

[7]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Vol. II , 1976 .

[8]  Jun Morimoto,et al.  Nonparametric Representation of Policies and Value Functions: A Trajectory-Based Approach , 2002, NIPS.

[9]  Charles J. Holland,et al.  A new energy characterization of the smallest eigenvalue of the schrödinger equation , 1977 .

[10]  Mark B. Milam,et al.  Real-Time Optimal Trajectory Generation for Constrained Dynamical Systems , 2003 .

[11]  Dani Lischinski,et al.  Target-driven smoke animation , 2004, SIGGRAPH 2004.

[12]  Joe Marks,et al.  Spacetime constraints revisited , 1993, SIGGRAPH.

[13]  Jovan Popovic,et al.  Simulation of Human Motion Data using Short‐Horizon Model‐Predictive Control , 2008, Comput. Graph. Forum.

[14]  Arun N. Netravali,et al.  Motion interpolation by optimal control , 1988, SIGGRAPH.

[15]  Jovan Popovic,et al.  Adaptation of performed ballistic motion , 2005, TOGS.

[16]  Petros Faloutsos,et al.  Composable controllers for physics-based character animation , 2001, SIGGRAPH.

[17]  David Q. Mayne,et al.  Differential dynamic programming , 1972, The Mathematical Gazette.

[18]  Pieter Abbeel,et al.  Apprenticeship learning via inverse reinforcement learning , 2004, ICML.

[19]  H. Kappen.  Linear theory for control of nonlinear stochastic systems. , 2004, Physical review letters.

[20]  Christopher G. Atkeson,et al.  Using Local Trajectory Optimizers to Speed Up Global Optimization in Dynamic Programming , 1993, NIPS.

[21]  Jessica K. Hodgins,et al.  Simulating leaping, tumbling, landing and balancing humans , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[22]  Eugene Fiume,et al.  Limit cycle control and its application to the animation of balancing and walking , 1996, SIGGRAPH.

[23]  William D. Smart,et al.  Bipedal walking on rough terrain using manifold control , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[24]  Jessica K. Hodgins,et al.  Motion capture-driven simulations that hit and react , 2002, SCA '02.

[25]  Andrew P. Witkin,et al.  Spacetime constraints , 1988, SIGGRAPH.

[26]  William D. Smart,et al.  Receding Horizon Differential Dynamic Programming , 2007, NIPS.

[27]  Marco da Silva,et al.  Interactive simulation of stylized human locomotion , 2008, ACM Trans. Graph..

[28]  Rajesh P. N. Rao,et al.  Bayesian brain : probabilistic approaches to neural coding , 2006 .

[29]  Bernt Øksendal,et al.  Stochastic differential equations (3rd ed.): an introduction with applications , 1992 .

[30]  Adrien Treuille,et al.  Near-optimal character animation with continuous control , 2007, SIGGRAPH 2007.

[31]  Jernej Barbič,et al.  Real-time control of physically based simulations using gentle forces , 2008, SIGGRAPH 2008.

[32]  Emanuel Todorov,et al.  Linearly-solvable Markov decision problems , 2006, NIPS.

[33]  Emanuel Todorov,et al.  Compositionality of optimal control laws , 2009, NIPS.

[34]  Eitan Grinspun,et al.  TRACKS: toward directable thin shells , 2007, SIGGRAPH 2007.

[35]  Nancy S. Pollard,et al.  Force-based motion editing for locomotion tasks , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[36]  Daniel E. Koditschek,et al.  Sequential Composition of Dynamically Dexterous Robot Behaviors , 1999, Int. J. Robotics Res..

[37]  W. Fleming.  Exit probabilities and optimal stochastic control , 1977 .

[38]  David C. Brogan,et al.  Animating human athletics , 1995, SIGGRAPH.

[39]  Z. Popovic,et al.  Fluid control using the adjoint method , 2004, SIGGRAPH 2004.

[40]  Jehee Lee,et al.  Simulating biped behaviors from human motion data , 2007, SIGGRAPH 2007.

[41]  Jessica K. Hodgins,et al.  Adapting simulated behaviors for new characters , 1997, SIGGRAPH.

[42]  M. V. D. Panne,et al.  SIMBICON: simple biped locomotion control , 2007, SIGGRAPH 2007.

[43]  Nancy S. Pollard,et al.  Responsive characters from motion fragments , 2007, SIGGRAPH 2007.

[44]  R. Bellman.  Dynamic programming. , 1957, Science.

[45]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[46]  Jessica K. Hodgins,et al.  Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces , 2004, SIGGRAPH 2004.