A Cerebellar Internal Models Control Architecture for Online Sensorimotor Adaptation of a Humanoid Robot Acting in a Dynamic Environment

Humanoid robots are often supposed to operate in non-deterministic human environments, and as a consequence, the robust and gentle rejection of the external perturbations is extremely crucial. In this scenario, stable and accurate behavior is mostly solved through adaptive control mechanisms that learn an internal model to predict the consequences of the outgoing control signals. Evidences show that brain-based biological systems resolve this control issue by updating an appropriate internal model that is then used to direct the muscles activities. Inspired by the biological cerebellar internal models theory, that couples forward and inverse internal models into the biological motor control scheme, we propose a novel methodology to artificially replicate these learning and adaptive principles into a robotic feedback controller. The proposed cerebellar-like network combines machine learning, artificial neural network, and computational neuroscience techniques to deal with all the nonlinearities and complexities that modern robotic systems could present. Although the architecture is tested on the simulated humanoid iCub, it can be applied to different robotic systems without excessive customization, thanks to its neural network-based nature. During the experiments, the robot is requested to follow repeatedly a movement while it is interacting with two external systems. Four different internal model architectures are compared and tested under different conditions. The comparison of the performances confirmed the theories about internal models combinatory action. The combination of models together with the structural and learning features of the network, resulted in a benefit to the adaptation mechanism, but also the system response to nonlinearities, noise and external forces.

[1]  Mitsuo Kawato,et al.  Feedback-Error-Learning Neural Network for Supervised Motor Learning , 1990 .

[2]  W. T. Miller,et al.  An overview of the CMAC neural network , 1991, [1991 Proceedings] IEEE Conference on Neural Networks for Ocean Engineering.

[3]  Kenji Doya,et al.  Neural mechanisms of learning and control , 2001 .

[4]  S P Wise,et al.  Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: their role in planning and controlling action. , 1995, Cerebral cortex.

[5]  G. Mettaa,et al.  A developmental approach to visually-guided reaching in artificial systems , 1999 .

[6]  Daniel M. Wolpert,et al.  Forward Models for Physiological Motor Control , 1996, Neural Networks.

[7]  Yian Deng,et al.  How Does a Robot Develop Its Reaching Ability Like Human Infants Do? , 2018, IEEE Transactions on Cognitive and Developmental Systems.

[8]  Hermano Igo Krebs,et al.  An Internal Model for Acquisition and Retention of Motor Learning During Arm Reaching , 2009, Neural Computation.

[9]  Silvia Tolu,et al.  A comprehensive gaze stabilization controller based on cerebellar internal models , 2017, Bioinspiration & biomimetics.

[10]  Chun-Yi Su,et al.  Adaptive Neural Network Control for Robotic Manipulators With Unknown Deadzone , 2018, IEEE Transactions on Cybernetics.

[11]  John Porrill,et al.  Recurrent Cerebellar Loops Simplify Adaptive Control of Redundant and Nonlinear Motor Systems , 2007, Neural Computation.

[12]  Eduardo Ros,et al.  Adaptive Robotic Control Driven by a Versatile Spiking Cerebellar Network , 2014, PloS one.

[13]  W. Thomas Miller,et al.  Sensor-based control of robotic manipulators using a general learning algorithm , 1987, IEEE J. Robotics Autom..

[14]  Eduardo Ros,et al.  Distributed Cerebellar Motor Learning: A Spike-Timing-Dependent Plasticity Model , 2016, Front. Comput. Neurosci..

[15]  P. Strick,et al.  Cerebellum and nonmotor function. , 2009, Annual review of neuroscience.

[16]  M. Kawato,et al.  A hierarchical neural-network model for control and learning of voluntary movement , 2004, Biological Cybernetics.

[17]  M. Arbib,et al.  Role of the cerebellum in reaching movements in humans. I. Distributed inverse dynamics control , 1998, The European journal of neuroscience.

[18]  Derong Liu,et al.  Adaptive approximation based control: Unifying neural, fuzzy and traditional adaptive approximation approaches. Jay A. Farrell and Marios M. Polycarpou, Wiley, New York, 2006. No of pages: 440. ISBN 978-0-471-72788-0 , 2008 .

[19]  Michael I. Jordan,et al.  Optimal feedback control as a theory of motor coordination , 2002, Nature Neuroscience.

[20]  A. G. Feldman New insights into action–perception coupling , 2009, Experimental Brain Research.

[21]  Jun Nakanishi,et al.  A Bayesian Approach to Nonlinear Parameter Identification for Rigid Body Dynamics , 2006, Robotics: Science and Systems.

[22]  Mitsuo Kawato,et al.  Feedback-error-learning neural network for trajectory control of a robotic manipulator , 1988, Neural Networks.

[23]  Giulio Sandini,et al.  A Survey of Artificial Cognitive Systems: Implications for the Autonomous Development of Mental Capabilities in Computational Agents , 2007, IEEE Transactions on Evolutionary Computation.

[24]  Silvia Tolu,et al.  Adaptive cerebellar Spiking Model Embedded in the Control Loop: Context Switching and Robustness against noise , 2011, Int. J. Neural Syst..

[25]  Laurentiu S. Popa,et al.  What Features of Limb Movements are Encoded in the Discharge of Cerebellar Neurons? , 2011, The Cerebellum.

[26]  M. Wiesendanger,et al.  Grip-load force coordination in cerebellar patients , 1999, Experimental Brain Research.

[27]  D M Wolpert,et al.  Multiple paired forward and inverse models for motor control , 1998, Neural Networks.

[28]  J. Albus A Theory of Cerebellar Function , 1971 .

[29]  G. Deuschl,et al.  Essential tremor and cerebellar dysfunction: abnormal ballistic movements , 2002, Journal of neurology, neurosurgery, and psychiatry.

[30]  Andreea C. Bostan,et al.  Consensus Paper: Towards a Systems-Level View of Cerebellar Function: the Interplay Between Cerebellum, Basal Ganglia, and Cortex , 2016, The Cerebellum.

[31]  Giulio Sandini,et al.  The iCub humanoid robot: an open platform for research in embodied cognition , 2008, PerMIS.

[32]  Ricardo O. Carelli,et al.  Neural networks for advanced control of robot manipulators , 2002, IEEE Trans. Neural Networks.

[33]  S. Scott The computational and neural basis of voluntary motor control and planning , 2012, Trends in Cognitive Sciences.

[34]  Richard B. Ivry,et al.  Consensus Paper: Roles of the Cerebellum in Motor Control—The Diversity of Ideas on Cerebellar Involvement in Movement , 2011, The Cerebellum.

[35]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[36]  Jean-René Duhamel,et al.  Optimal Sensorimotor Control in Eye Movement Sequences , 2009, The Journal of Neuroscience.

[37]  Snehasis Mukhopadhyay,et al.  Associative learning in random environments using neural networks , 1991, IEEE Trans. Neural Networks.

[38]  Timothy J. Ebner,et al.  Purkinje Cell Simple Spike Discharge Encodes Error Signals Consistent with a Forward Internal Model , 2013, The Cerebellum.

[39]  Soichi Nagao,et al.  Tandem internal models execute motor learning in the cerebellum , 2018, Proceedings of the National Academy of Sciences.

[40]  Eduardo Ros,et al.  Modeling the Cerebellar Microcircuit: New Strategies for a Long-Standing Issue , 2016, Front. Cell. Neurosci..

[41]  Jun Nakanishi,et al.  Feedback error learning and nonlinear adaptive control , 2004, Neural Networks.

[42]  Joachim Hermsdörfer,et al.  The role of the cerebellum for predictive control of grasping , 2008, The Cerebellum.

[43]  Jan Peters,et al.  Model learning for robot control: a survey , 2011, Cognitive Processing.

[44]  Shuzhi Sam Ge,et al.  Adaptive neural network control for strict-feedback nonlinear systems using backstepping design , 1999, Proceedings of the 1999 American Control Conference (Cat. No. 99CH36251).

[45]  James C. Houk,et al.  A Cerebellar Model of Timing and Prediction in the Control of Reaching , 1999, Neural Computation.

[46]  Silvia Tolu,et al.  Bio-inspired adaptive feedback error learning architecture for motor control , 2012, Biological Cybernetics.

[47]  G. Pezzulo,et al.  Neuroscience and Biobehavioral Reviews the Contribution of Brain Sub-cortical Loops in the Expression and Acquisition of Action Understanding Abilities , 2022 .

[48]  Stefan Ulbrich,et al.  A Framework for Coupled Simulations of Robots and Spiking Neuronal Networks , 2016, J. Intell. Robotic Syst..

[49]  Mark J Wagner,et al.  Shared Internal Models for Feedforward and Feedback Control , 2008, The Journal of Neuroscience.

[50]  Jun Nakanishi,et al.  Operational Space Control: A Theoretical and Empirical Comparison , 2008, Int. J. Robotics Res..

[51]  D. Wolpert,et al.  Is the cerebellum a smith predictor? , 1993, Journal of motor behavior.

[52]  J. Farrell,et al.  Adaptive Approximation Based Control: Unifying Neural, Fuzzy and Traditional Adaptive Approximation Approaches (Adaptive and Learning Systems for Signal Processing, Communications and Control Series) , 2006 .

[53]  Rüdiger Dillmann,et al.  Connecting Artificial Brains to Robots in a Comprehensive Simulation Framework: The Neurorobotics Platform , 2017, Front. Neurorobot..

[54]  Anthony Leonardo,et al.  Internal models direct dragonfly interception steering , 2014, Nature.

[55]  K. Narendra,et al.  Persistent excitation in adaptive systems , 1987 .

[56]  Christopher G. Atkeson,et al.  Estimation of Inertial Parameters of Manipulator Loads and Links , 1986 .

[57]  Egidio D'Angelo,et al.  Distributed cerebellar plasticity implements adaptable gain control in a manipulation task: a closed-loop robotic simulation , 2013, Front. Neural Circuits.

[58]  Mitsuo Kawato,et al.  Internal models for motor control and trajectory planning , 1999, Current Opinion in Neurobiology.

[59]  Stefan Schaal,et al.  Locally Weighted Projection Regression : An O(n) Algorithm for Incremental Real Time Learning in High Dimensional Space , 2000 .

[60]  D. Marr A theory of cerebellar cortex , 1969, The Journal of physiology.

[61]  Robert M. Sanner,et al.  Gaussian Networks for Direct Adaptive Control , 1991, 1991 American Control Conference.

[62]  Hiroshi Imamizu,et al.  Human cerebellar activity reflecting an acquired internal model of a new tool , 2000, Nature.

[63]  D. Wolpert,et al.  Internal models in the cerebellum , 1998, Trends in Cognitive Sciences.

[64]  Wey-Shiuan Hwang,et al.  The Developmental Approach to Artiicial Intelligence: Concepts, Developmental Algorithms and Experimental Results , 1999 .

[65]  J. Kocijan,et al.  Gaussian process model based predictive control , 2004, Proceedings of the 2004 American Control Conference.

[66]  Tatsuya Kimura,et al.  Cerebellar complex spikes encode both destinations and errors in arm movements , 1998, Nature.

[67]  Masao Ito The Cerebellum And Neural Control , 1984 .

[68]  Chris Eliasmith,et al.  A spiking neural model of adaptive arm control , 2016, Proceedings of the Royal Society B: Biological Sciences.

[69]  J. Medina,et al.  The multiple roles of Purkinje cells in sensori-motor calibration: to predict, teach and command , 2011, Current Opinion in Neurobiology.

[70]  Karl J. Friston What Is Optimal about Motor Control? , 2011, Neuron.

[71]  Michael I. Jordan,et al.  Forward Models: Supervised Learning with a Distal Teacher , 1992, Cogn. Sci..

[72]  Bruno Lara-Guzmán,et al.  A Self-Organized Internal Models Architecture for Coding Sensory–Motor Schemes , 2016, Front. Robot. AI.

[73]  James S. Albus,et al.  New Approach to Manipulator Control: The Cerebellar Model Articulation Controller (CMAC)1 , 1975 .

[74]  Jesús Alberto Garrido,et al.  Integrated neural and robotic simulations. Simulation of cerebellar neurobiological substrate for an object-oriented dynamic model abstraction process , 2014, Robotics Auton. Syst..

[75]  Silvia Tolu,et al.  Adaptive and Predictive Control of a Simulated Robot arm , 2013, Int. J. Neural Syst..

[76]  Dana Chidekel,et al.  From Movement to Thought: Executive Function, Embodied Cognition, and the Cerebellum , 2012, Cerebellum.

[77]  Eduardo Ros,et al.  Fast convergence of learning requires plasticity between inferior olive and deep cerebellar nuclei in a manipulation task: a closed-loop robotic simulation , 2014, Front. Comput. Neurosci..

[78]  J. Simpson,et al.  Microcircuitry and function of the inferior olive , 1998, Trends in Neurosciences.