Hierarchical motor control in mammals and machines

Advances in artificial intelligence are stimulating interest in neuroscience. However, most attention is given to discrete tasks with simple action spaces, such as board games and classic video games. Less discussed in neuroscience are parallel advances in “synthetic motor control”. While motor neuroscience has recently focused on optimization of single, simple movements, AI has progressed to the generation of rich, diverse motor behaviors across multiple tasks, at humanoid scale. It is becoming clear that specific, well-motivated hierarchical design elements repeatedly arise when engineering these flexible control systems. We review these core principles of hierarchical control, relate them to hierarchy in the nervous system, and highlight research themes that we anticipate will be critical in solving challenges at this disciplinary intersection. Recent research in motor neuroscience has focused on optimal feedback control of single, simple tasks while robotics and AI are making progress towards flexible movement control in complex environments employing hierarchical control strategies. Here, the authors argue for a return to hierarchical models of motor control in neuroscience.

[1]  K. Lashley Basic neural mechanisms in behavior. , 1930 .

[2]  E. Culler,et al.  Conditioned behavior in a decorticate dog. , 1934 .

[3]  R. McK. Certain Aspects of the Behavior of Decorticate Cats , 1938 .

[4]  William Rowan,et al.  The Study of Instinct , 1953 .

[5]  H. Evans The Study of Instinct , 1952 .

[6]  Marvin Minsky,et al.  Steps toward Artificial Intelligence , 1995, Proceedings of the IRE.

[7]  R. Bellman Dynamic programming. , 1957, Science.

[8]  M. Wayner Motor control functions of the lateral hypothalamus and adjunctive behavior. , 1970, Physiology & behavior.

[9]  CHANNELING OF RESPONSES ELICITED BY HYPOTHALAMIC STIMULATION , 1972 .

[10]  R. Wise Lateral hypothalamic electrical stimulation: does it make animals 'hungry'? , 1974, Brain research.

[11]  J. Tanji,et al.  Anticipatory activity of motor cortex neurons in relation to direction of an intended movement. , 1976, Journal of neurophysiology.

[12]  M. Raibert Motor Control and Learning by the State Space Model , 1977 .

[13]  Douglas L. Jones,et al.  From motivation to action: Functional interface between the limbic system and the motor system , 1980, Progress in Neurobiology.

[14]  S. Grillner Neurobiological bases of rhythmic motor acts in vertebrates. , 1985, Science.

[15]  T. Flash,et al.  The coordination of arm movements: an experimentally confirmed mathematical model , 1985, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[16]  I. Whishaw,et al.  The mating movements of male decorticate rats: Evidence for subcortically generated movements by the male but regulation of approaches by the female , 1985, Behavioural Brain Research.

[17]  Rodney A. Brooks,et al.  A Robust Layered Control Syste For A Mobile Robot , 2022 .

[18]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .

[19]  F A Mussa-Ivaldi,et al.  Computations underlying the execution of movement: a biological perspective. , 1991, Science.

[20]  Geoffrey E. Hinton,et al.  Feudal Reinforcement Learning , 1992, NIPS.

[21]  M. Udo,et al.  A new learning paradigm: adaptive changes in interlimb coordination during perturbed locomotion in decerebrate cats , 1993, Neuroscience Research.

[22]  F. A. Mussa-lvaldi,et al.  Convergent force fields organized in the frog's spinal cord , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[23]  Richard S. Sutton,et al.  A Menu of Designs for Reinforcement Learning Over Time , 1995 .

[24]  Örjan Ekeberg,et al.  The Neural Control of Fish Swimming Studied Through Numerical Simulations , 1995, Adapt. Behav..

[25]  Ivan Bratko,et al.  Behavioural Cloning: Phenomena, Results and Problems , 1995 .

[26]  Joel L. Davis,et al.  In : Models of Information Processing in the Basal Ganglia , 2008 .

[27]  P. Whelan CONTROL OF LOCOMOTION IN THE DECEREBRATE CAT , 1996, Progress in Neurobiology.

[28]  N. A. Bernstein Dexterity and Its Development , 1996 .

[29]  Daniel M. Wolpert,et al.  Forward Models for Physiological Motor Control , 1996, Neural Networks.

[30]  Michael I. Jordan Chapter 2 Computational aspects of motor control and motor learning , 1996 .

[31]  Richard S. J. Frackowiak,et al.  Anatomy of motor learning. II. Subcortical structures and learning by trial and error. , 1997, Journal of neurophysiology.

[32]  Geoffrey E. Hinton,et al.  NeuroAnimator: fast neural network emulation and control of physics-based models , 1998, SIGGRAPH.

[33]  I. Whishaw,et al.  Paw and limb use in skilled and spontaneous reaching after pyramidal tract, red nucleus and combined lesions in the rat: behavioral and anatomical dissociations , 1998, Behavioural Brain Research.

[34]  Mitsuo Kawato,et al.  Internal models for motor control and trajectory planning , 1999, Current Opinion in Neurobiology.

[35]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[36]  Peter Redgrave,et al.  Layered Control Architectures in Robots and Vertebrates , 1999, Adapt. Behav..

[37]  D. Rose Progress in Motor Control: Bernstein's Traditions in Movement Studies, Volume 1 , 1999 .

[38]  M. Tomasello The Cultural Origins of Human Cognition , 2000 .

[39]  L. Swanson Cerebral hemisphere regulation of motivated behavior 1 1 Published on the World Wide Web on 2 November 2000. , 2000, Brain Research.

[40]  Zoubin Ghahramani,et al.  Computational principles of movement neuroscience , 2000, Nature Neuroscience.

[41]  E. Bizzi,et al.  New perspectives on spinal motor systems , 2000, Nature Reviews Neuroscience.

[42]  E. Marder,et al.  Central pattern generators and the control of rhythmic movements , 2001, Current Biology.

[43]  G. Rizzolatti,et al.  Neurophysiological mechanisms underlying the understanding and imitation of action , 2001, Nature Reviews Neuroscience.

[44]  Petros Faloutsos,et al.  Composable controllers for physics-based character animation , 2001, SIGGRAPH.

[45]  Michael I. Jordan,et al.  Optimal feedback control as a theory of motor coordination , 2002, Nature Neuroscience.

[46]  M. Graziano,et al.  Complex Movements Evoked by Microstimulation of Precentral Cortex , 2002, Neuron.

[47]  Sridhar Mahadevan,et al.  Recent Advances in Hierarchical Reinforcement Learning , 2003, Discret. Event Dyn. Syst..

[48]  Zoubin Ghahramani,et al.  Unsupervised learning of sensory-motor primitives , 2003, Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No.03CH37439).

[49]  V. Dietz Spinal cord pattern generators for locomotion , 2003, Clinical Neurophysiology.

[50]  Miomir Vukobratovic,et al.  Zero-Moment Point - Thirty Five Years of its Life , 2004, Int. J. Humanoid Robotics.

[51]  S. Grillner,et al.  On the central generation of locomotion in the low spinal cat , 1979, Experimental Brain Research.

[52]  E. Todorov Optimality principles in sensorimotor control , 2004, Nature Neuroscience.

[53]  Emanuel Todorov,et al.  From task parameters to motor synergies: A hierarchical framework for approximately optimal control of redundant manipulators , 2005 .

[54]  Emanuel Todorov,et al.  From task parameters to motor synergies: A hierarchical framework for approximately optimal control of redundant manipulators , 2005, J. Field Robotics.

[55]  Scott L. Delp,et al.  A Model of the Upper Extremity for Simulating Musculoskeletal Surgery and Analyzing Neuromuscular Control , 2005, Annals of Biomedical Engineering.

[56]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[57]  Aaron S. Andalman,et al.  Vocal Experimentation in the Juvenile Songbird Requires a Basal Ganglia Circuit , 2005, PLoS biology.

[58]  K. Pearson,et al.  Computer simulation of stepping in the hind legs of the cat: an examination of mechanisms regulating the stance-to-swing transition. , 2005, Journal of neurophysiology.

[59]  Byron M. Yu,et al.  Neural Variability in Premotor Cortex Provides a Signature of Motor Preparation , 2006, The Journal of Neuroscience.

[60]  Rajesh P. N. Rao,et al.  Bayesian brain : probabilistic approaches to neural coding , 2006 .

[61]  Emanuel Todorov,et al.  Optimal Control Theory , 2006 .

[62]  George K York,et al.  An introduction to the life and work of John Hughlings Jackson with a catalogue raisonné of his writings. , 2006, Medical history. Supplement.

[63]  J. Jackson An Introduction to the Life and Work of John Hughlings Jackson , 2006, Medical History.

[64]  A. Ijspeert,et al.  From Swimming to Walking with a Salamander Robot Driven by a Spinal Cord Model , 2007, Science.

[65]  G. Csibra,et al.  'Obsessed with goals': functions and mechanisms of teleological interpretation of actions in humans. , 2007, Acta psychologica.

[66]  M. V. D. Panne,et al.  SIMBICON: simple biped locomotion control , 2007, SIGGRAPH 2007.

[67]  Auke Jan Ijspeert,et al.  Central pattern generators for locomotion control in animals and robots: A review , 2008, Neural Networks.

[68]  S. Grillner,et al.  Neural bases of goal-directed locomotion in vertebrates—An overview , 2008, Brain Research Reviews.

[69]  David Badre,et al.  Cognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes , 2008, Trends in Cognitive Sciences.

[70]  Michale S Fee,et al.  A Specialized Forebrain Circuit for Vocal Babbling in the Juvenile Songbird , 2008, Science.

[71]  R. Lemon Descending pathways in motor control. , 2008, Annual review of neuroscience.

[72]  Michael I. Jordan Computational aspects of motor control and motor learning , 2008 .

[73]  M. Tomasello Origins of human communication , 2008 .

[74]  Scott L. Delp,et al.  A Model of the Lower Limb for Analysis of Human Movement , 2010, Annals of Biomedical Engineering.

[75]  M. Botvinick,et al.  Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective , 2009, Cognition.

[76]  Hillel J. Chiel,et al.  The Brain in Its Body: Motor Control and Sensing in a Biomechanical Context , 2009, The Journal of Neuroscience.

[77]  J. Andrew Bagnell,et al.  Efficient Reductions for Imitation Learning , 2010, AISTATS.

[78]  Aaron Hertzmann,et al.  Robust physics-based locomotion using low-dimensional planning , 2010, SIGGRAPH 2010.

[79]  Benjamin O. Turner,et al.  Cortical and basal ganglia contributions to habit learning and automaticity , 2010, Trends in Cognitive Sciences.

[80]  R. Ivry,et al.  The coordination of movement: optimal feedback control and beyond , 2010, Trends in Cognitive Sciences.

[81]  A. Karpathy,et al.  Locomotion skills for simulated quadrupeds , 2011, SIGGRAPH 2011.

[82]  A. d’Avella,et al.  Locomotor Primitives in Newborn Babies and Their Development , 2011, Science.

[83]  G. Csibra,et al.  Natural pedagogy as evolutionary adaptation , 2011, Philosophical Transactions of the Royal Society B: Biological Sciences.

[84]  Karl J. Friston What Is Optimal about Motor Control? , 2011, Neuron.

[85]  Theresa J Klein,et al.  A physical model of sensorimotor interactions during locomotion , 2012, Journal of neural engineering.

[86]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[87]  Trevor Bekolay,et al.  Supplementary Materials for A Large-Scale Model of the Functioning Brain , 2012 .

[88]  Yuval Tassa,et al.  Synthesis and stabilization of complex behaviors through online trajectory optimization , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[89]  Baining Guo,et al.  Terrain runner , 2012, ACM Trans. Graph..

[90]  Yuval Tassa,et al.  MuJoCo: A physics engine for model-based control , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[91]  Zoran Popovic,et al.  Discovery of complex behaviors through contact-invariant optimization , 2012, ACM Trans. Graph..

[92]  S. Scott The computational and neural basis of voluntary motor control and planning , 2012, Trends in Cognitive Sciences.

[93]  Sergey Levine,et al.  Guided Policy Search , 2013, ICML.

[94]  Matthew Millard,et al.  Flexing computational muscle: modeling and simulation of musculotendon dynamics. , 2013, Journal of biomechanical engineering.

[95]  Vladlen Koltun,et al.  Animating human lower limbs using contact-invariant optimization , 2013, ACM Trans. Graph..

[96]  S. Sternson Hypothalamic Survival Circuits: Blueprints for Purposive Behaviors , 2013, Neuron.

[97]  L. Bonazzi,et al.  Complex Movement Topography and Extrinsic Space Representation in the Rat Forelimb Motor Cortex as Defined by Long-Duration Intracortical Microstimulation , 2013, The Journal of Neuroscience.

[98]  Hugo Merchant,et al.  Motor system evolution and the emergence of high cognitive functions , 2014, Progress in Neurobiology.

[99]  David J. Anderson,et al.  Decoding Ventromedial Hypothalamic Neural Activity during Male Mouse Aggression , 2014, The Journal of Neuroscience.

[100]  Emanuel Todorov,et al.  Combining the benefits of function approximation and trajectory optimization , 2014, Robotics: Science and Systems.

[101]  Andrew R. Brown,et al.  Motor Cortex Is Functionally Organized as a Set of Spatially Distinct Representations for Complex Movements , 2014, The Journal of Neuroscience.

[102]  L. F. Abbott,et al.  Hierarchical Control Using Networks Trained with Higher-Level Forward Models , 2014, Neural Computation.

[103]  Matthew T. Kaufman,et al.  Supplementary materials for : Cortical activity in the null space : permitting preparation without movement , 2014 .

[104]  Zengcai V. Guo,et al.  Flow of Cortical Activity Underlying a Tactile Decision in Mice , 2014, Neuron.

[105]  Richard Hans Robert Hahnloser,et al.  Evidence for a causal inverse model in an avian cortico-basal ganglia circuit , 2014, Proceedings of the National Academy of Sciences.

[106]  Ashesh K Dhawale,et al.  Motor Cortex Is Required for Learning but Not for Executing a Motor Skill , 2015, Neuron.

[107]  Scott T. Grafton,et al.  The striatum: where skills and habits meet. , 2015, Cold Spring Harbor perspectives in biology.

[108]  Zengcai V. Guo,et al.  A motor cortex circuit for motor planning and movement , 2015, Nature.

[109]  Zoran Popovic,et al.  Interactive Control of Diverse Complex Characters with Neural Networks , 2015, NIPS.

[110]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[111]  Stefano Ermon,et al.  Generative Adversarial Imitation Learning , 2016, NIPS.

[112]  Sergey Levine,et al.  End-to-End Training of Deep Visuomotor Policies , 2015, J. Mach. Learn. Res..

[113]  Sergey Levine,et al.  High-Dimensional Continuous Control Using Generalized Advantage Estimation , 2015, ICLR.

[114]  Joshua B. Tenenbaum,et al.  Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation , 2016, NIPS.

[115]  Yuval Tassa,et al.  Learning and Transfer of Modulated Locomotor Controllers , 2016, ArXiv.

[116]  Tom Schaul,et al.  FeUdal Networks for Hierarchical Reinforcement Learning , 2017, ICML.

[117]  J. Hodgins,et al.  Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning , 2017 .

[118]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[119]  Ryan Remedios,et al.  Social Behaviour Shapes Hypothalamic Neural Ensemble Representations Of Conspecific Sex , 2017, Nature.

[120]  Marcin Andrychowicz,et al.  One-Shot Imitation Learning , 2017, NIPS.

[121]  Xiao-Jing Wang,et al.  Reward-based training of recurrent neural networks for cognitive and value-based tasks , 2016, bioRxiv.

[122]  Joseph J. Paton,et al.  A robust role for motor cortex , 2016, bioRxiv.

[123]  Matthew T. Kaufman,et al.  Perspectives on classical controversies about the motor cortex. , 2017, Journal of neurophysiology.

[124]  Glen Berseth,et al.  DeepLoco: dynamic locomotion skills using hierarchical deep reinforcement learning , 2017, ACM Trans. Graph..

[125]  Yuval Tassa,et al.  Emergence of Locomotion Behaviours in Rich Environments , 2017, ArXiv.

[126]  Pietro Perona,et al.  Learning recurrent representations for hierarchical behavior modeling , 2016, ICLR.

[127]  M. A. MacIver,et al.  Neuroscience Needs Behavior: Correcting a Reductionist Bias , 2017, Neuron.

[128]  Nando de Freitas,et al.  Robust Imitation of Diverse Behaviors , 2017, NIPS.

[129]  Yee Whye Teh,et al.  Distral: Robust multitask reinforcement learning , 2017, NIPS.

[130]  Tom Schaul,et al.  Reinforcement Learning with Unsupervised Auxiliary Tasks , 2016, ICLR.

[131]  Yuval Tassa,et al.  Learning human behaviors from motion capture by adversarial imitation , 2017, ArXiv.

[132]  Georg B. Keller,et al.  Mouse Motor Cortex Coordinates the Behavioral Response to Unpredicted Sensory Feedback , 2018, Neuron.

[133]  Sergey Levine,et al.  DeepMimic , 2018, ACM Trans. Graph..

[134]  K. Svoboda,et al.  Neural mechanisms of movement planning: motor cortex and beyond , 2018, Current Opinion in Neurobiology.

[135]  Scott W. Linderman,et al.  The Striatum Organizes 3D Behavior via Moment-to-Moment Action Selection , 2018, Cell.

[136]  Aaron C. Courville,et al.  FiLM: Visual Reasoning with a General Conditioning Layer , 2017, AAAI.

[137]  Yuval Tassa,et al.  DeepMind Control Suite , 2018, ArXiv.

[138]  Oriol Vinyals,et al.  Representation Learning with Contrastive Predictive Coding , 2018, ArXiv.

[139]  Michaela Bruton,et al.  Synergies in coordination: a comprehensive overview of neural, computational, and behavioral approaches. , 2018, Journal of neurophysiology.

[140]  Joel Z. Leibo,et al.  Unsupervised Predictive Memory in a Goal-Directed Agent , 2018, ArXiv.

[141]  Nando de Freitas,et al.  Reinforcement and Imitation Learning for Diverse Visuomotor Skills , 2018, Robotics: Science and Systems.

[142]  Pierre-Yves Oudeyer,et al.  Towards a neuroscience of active sampling and curiosity , 2018, Nature Reviews Neuroscience.

[143]  Yee Whye Teh,et al.  Information asymmetry in KL-regularized RL , 2019, ICLR.

[144]  Scott W. Linderman,et al.  Probabilistic Models of Larval Zebrafish Behavior: Structure on Many Scales , 2019, bioRxiv.

[145]  Scott W. Linderman,et al.  Hierarchical recurrent state space models reveal discrete and continuous dynamics of neural activity in C. elegans , 2019, bioRxiv.

[146]  Sergey Levine,et al.  Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow , 2018, ICLR.

[147]  Yee Whye Teh,et al.  Neural probabilistic motor primitives for humanoid control , 2018, ICLR.

[148]  Nicolas Heess,et al.  Hierarchical visuomotor control of humanoids , 2018, ICLR.

[149]  Joonho Lee,et al.  Learning agile and dynamic motor skills for legged robots , 2019, Science Robotics.

[150]  Sergey Levine,et al.  Near-Optimal Representation Learning for Hierarchical Reinforcement Learning , 2018, ICLR.

[151]  P. Cisek Resynthesizing behavior through phylogenetic refinement , 2019, Attention, Perception, & Psychophysics.

[152]  Mark L. Latash,et al.  On the Construction of Movements , 2020 .

[153]  Jakub W. Pachocki,et al.  Learning dexterous in-hand manipulation , 2018, Int. J. Robotics Res..