Learned parametrized dynamic movement primitives with shared synergies for controlling robotic and musculoskeletal systems

A salient feature of human motor skill learning is the ability to exploit similarities across related tasks. In biological motor control, it has been hypothesized that muscle synergies, coherent activations of groups of muscles, allow for exploiting shared knowledge. Recent studies have shown that a rich set of complex motor skills can be generated by a combination of a small number of muscle synergies. In robotics, dynamic movement primitives are commonly used for motor skill learning. This machine learning approach implements a stable attractor system that facilitates learning and it can be used in high-dimensional continuous spaces. However, it does not allow for reusing shared knowledge, i.e., for each task an individual set of parameters has to be learned. We propose a novel movement primitive representation that employs parametrized basis functions, which combines the benefits of muscle synergies and dynamic movement primitives. For each task a superposition of synergies modulates a stable attractor system. This approach leads to a compact representation of multiple motor skills and at the same time enables efficient learning in high-dimensional continuous systems. The movement representation supports discrete and rhythmic movements and in particular includes the dynamic movement primitive approach as a special case. We demonstrate the feasibility of the movement representation in three multi-task learning simulated scenarios. First, the characteristics of the proposed representation are illustrated in a point-mass task. Second, in complex humanoid walking experiments, multiple walking patterns with different step heights are learned robustly and efficiently. Finally, in a multi-directional reaching task simulated with a musculoskeletal model of the human arm, we show how the proposed movement primitives can be used to learn appropriate muscle excitation patterns and to generalize effectively to new reaching skills.

[1]  Simon A. Overduin,et al.  Modulation of Muscle Synergy Recruitment in Primate Grasping , 2008, The Journal of Neuroscience.

[2]  H. Christensen.  EMG-analysis , 1989, Images of the Twenty-First Century. Proceedings of the Annual International Engineering in Medicine and Biology Society,.

[3]  Antonie J. van den Bogert,et al.  A Real-Time, 3-D Musculoskeletal Model for Dynamic Simulation of Arm Movements , 2009, IEEE Transactions on Biomedical Engineering.

[4]  Petros Koumoutsakos,et al.  Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[5]  Jack M. Winters,et al.  Analysis of Fundamental Human Movement Patterns Through the Use of In-Depth Antagonistic Muscle Models , 1985, IEEE Transactions on Biomedical Engineering.

[6]  Francesco Lacquaniti,et al.  Control of Fast-Reaching Movements by Muscle Synergy Combinations , 2006, The Journal of Neuroscience.

[7]  Emilio Bizzi,et al.  Shared and specific muscle synergies in natural motor behaviors. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[8]  Emilio Bizzi,et al.  Combinations of muscle synergies in the construction of a natural motor behavior , 2003, Nature Neuroscience.

[9]  E. Bizzi,et al.  Article history: , 2005 .

[10]  Richard R Neptune,et al.  Modular control of human walking: a simulation study. , 2009, Journal of biomechanics.

[11]  Stefan Schaal,et al.  Movement segmentation using a primitive library , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  F. Zajac.  Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. , 1989, Critical reviews in biomedical engineering.

[13]  R. Angel,et al.  Electromyographic patterns during ballistic movement of normal and spastic limbs , 1975, Brain Research.

[14]  Andrea d'Avella,et al.  Synthesis and Adaptation of Effective Motor Synergies for the Solution of Reaching Tasks , 2012, SAB.

[15]  Albert Mukovskiy,et al.  Real-Time Synthesis of Body Movements Based on Learned Primitives , 2009, Statistical and Geometrical Approaches to Visual Motion Analysis.

[16]  Jeffrey A Reinbolt,et al.  OpenSim: a musculoskeletal modeling and simulation framework for in silico investigations and exchange. , 2011, Procedia IUTAM.

[17]  Jan Peters,et al.  Reinforcement Learning to Adjust Robot Movements to New Situations , 2010, IJCAI.

[18]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[19]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[20]  Lena H. Ting,et al.  Optimization of Muscle Activity for Task-Level Goals Predicts Complex Changes in Limb Forces across Biomechanical Contexts , 2012, PLoS Comput. Biol..

[21]  Jessy W. Grizzle,et al.  Experimental Validation of a Framework for the Design of Controllers that Induce Stable Walking in Planar Bipeds , 2004, Int. J. Robotics Res..

[22]  Ayman Habib,et al.  OpenSim: Open-Source Software to Create and Analyze Dynamic Simulations of Movement , 2007, IEEE Transactions on Biomedical Engineering.

[23]  R. M. Glaser,et al.  Improving the efficacy of electrical stimulation-induced leg cycle ergometry: an analysis based on a dynamic musculoskeletal model , 1993 .

[24]  Jun Nakanishi,et al.  Learning Attractor Landscapes for Learning Motor Primitives , 2002, NIPS.

[25]  D. B. Lockhart,et al.  Optimal sensorimotor transformations for balance , 2007, Nature Neuroscience.

[26]  Scott L. Delp,et al.  A Model of the Upper Extremity for Simulating Musculoskeletal Surgery and Analyzing Neuromuscular Control , 2005, Annals of Biomedical Engineering.

[27]  Andrea d'Avella,et al.  Modularity in the motor system: decomposition of muscle patterns as combinations of time-varying synergies , 2001, NIPS.

[28]  Oliver Kroemer,et al.  Learning to select and generalize striking movements in robot table tennis , 2012, AAAI Fall Symposium: Robots Learning Interactively from Human Teachers.

[29]  Tom Schaul,et al.  Episodic Reinforcement Learning by Logistic Reward-Weighted Regression , 2008, ICANN.

[30]  M. Hallett,et al.  EMG analysis of stereotyped voluntary movements in man. , 1975, Journal of neurology, neurosurgery, and psychiatry.

[31]  Jun Nakanishi,et al.  Learning Movement Primitives , 2005, ISRR.

[32]  Emanuel Todorov,et al.  Iterative Linear Quadratic Regulator Design for Nonlinear Biological Movement Systems , 2004, ICINCO.

[33]  Stefan Schaal,et al.  Reinforcement learning of motor skills in high dimensions: A path integral approach , 2010, 2010 IEEE International Conference on Robotics and Automation.

[34]  Stefan Schaal,et al.  2008 Special Issue: Reinforcement learning of motor skills with policy gradients , 2008 .

[35]  Jun Morimoto,et al.  Learning from demonstration and adaptation of biped locomotion , 2004, Robotics Auton. Syst..

[36]  Anthony Jarc,et al.  Simplified and effective motor control based on muscle synergies to exploit musculoskeletal dynamics , 2009, Proceedings of the National Academy of Sciences.

[37]  Jan Peters,et al.  Noname manuscript No. (will be inserted by the editor) Policy Search for Motor Primitives in Robotics , 2022 .

[38]  T. Pozzo,et al.  Tri-dimensional and triphasic muscle organization of whole-body pointing movements , 2010, Neuroscience.

[39]  Stefan Schaal,et al.  Learning and generalization of motor skills by learning from demonstration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[40]  A. d’Avella,et al.  Locomotor Primitives in Newborn Babies and Their Development , 2011, Science.

[41]  M G Pandy,et al.  Musculoskeletal Model of the Upper Limb Based on the Visible Human Male Dataset , 2001, Computer methods in biomechanics and biomedical engineering.

[42]  F. Lacquaniti,et al.  Five basic muscle activation patterns account for muscle activity during human locomotion , 2004, The Journal of physiology.

[43]  Jun Nakanishi,et al.  Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors , 2013, Neural Computation.

[44]  Marc Toussaint,et al.  Learned graphical models for probabilistic planning provide a new class of movement primitives , 2013, Front. Comput. Neurosci..

[45]  Walter Herzog,et al.  Model-based estimation of muscle forces exerted during movements. , 2007, Clinical biomechanics.

[46]  Robert A. Jacobs,et al.  Properties of Synergies Arising from a Theory of Optimal Motor Behavior , 2006, Neural Computation.

[47]  Frank Sehnke,et al.  Parameter-exploring policy gradients , 2010, Neural Networks.

[48]  M. Hallett,et al.  Single-joint rapid arm movements in normal subjects and in patients with motor disorders. , 1996, Brain : a journal of neurology.

[49]  Andrea d'Avella,et al.  Modularity for Sensorimotor Control: Evidence and a New Prediction , 2010, Journal of motor behavior.