Application of the Actor-Critic Architecture to Functional Electrical Stimulation Control of a Human Arm

Clinical tests have shown that the dynamics of a human arm, controlled using Functional Electrical Stimulation (FES), can vary significantly between and during trials. In this paper, we study the application of the actor-critic architecture, with neural networks for the both the actor and the critic, as a controller that can adapt to these changing dynamics of a human arm. Development and tests were done in simulation using a planar arm model and Hill-based muscle dynamics. We begin by training it using a Proportional Derivative (PD) controller as a supervisor. We then make clinically relevant changes to the dynamics of the arm and test the actor-critic's ability to adapt without supervision in a reasonable number of episodes. Finally, we devise methods for achieving both rapid learning and long-term stability.

[1]  H.J. Chizeck,et al.  Feedback regulation of hand grasp opening and contact force during stimulation of paralyzed muscle , 1991, IEEE Transactions on Biomedical Engineering.

[2]  A. Schultz,et al.  A simple Hill element-nonlinear spring model of muscle contraction biomechanics. , 1991, Journal of applied physiology.

[3]  R. Jaeger,et al.  Lower extremity applications of functional neuromuscular stimulation. , 1992, Assistive technology : the official journal of RESNA.

[4]  E. Marsolais,et al.  Synthesis of paraplegic gait with multichannel functional neuromuscular stimulation , 1994 .

[5]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[6]  J J Abbas,et al.  New control strategies for neuroprosthetic systems. , 1996, Journal of rehabilitation research and development.

[7]  J J Abbas,et al.  Experimental evaluation of an adaptive feedforward controller for use in functional neuromuscular stimulation systems. , 1993, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[8]  T S Kuo,et al.  A neuro-control system for the knee joint position control with quadriceps stimulation. , 1997, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[9]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[10]  B J Andrews,et al.  Computer simulation of FES standing up in paraplegia: a self-adaptive fuzzy controller with reinforcement learning. , 1998, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[11]  R. Stein,et al.  Functional electrical stimulation after spinal cord injury. , 1999, Journal of neurotrauma.

[12]  Kenji Doya,et al.  Reinforcement Learning in Continuous Time and Space , 2000, Neural Computation.

[13]  K. Kilgore,et al.  Efficacy of an implanted neuroprosthesis for restoring hand grasp in tetraplegia: a multicenter study. , 2001, Archives of physical medicine and rehabilitation.

[14]  B. Andrews,et al.  Functional electric stimulation-assisted rowing: Increasing cardiovascular fitness through functional electric stimulation rowing training in persons with spinal cord injury. , 2002, Archives of physical medicine and rehabilitation.

[15]  S. McLean,et al.  Development and validation of a 3-D model to predict knee joint loading during dynamic movement. , 2003, Journal of biomechanical engineering.

[16]  Sybert H. Stroeve,et al.  Learning combined feedback and feedforward control of a musculoskeletal system , 1996, Biological Cybernetics.

[17]  Stefan Schaal,et al.  Scalable Techniques from Nonparametric Statistics for Real Time Robot Learning , 2002, Applied Intelligence.

[18]  Toshiyuki Kondo,et al.  Biological arm motion through reinforcement learning , 2004, Biological Cybernetics.

[19]  P. Peckham,et al.  Functional electrical stimulation for neuromuscular applications. , 2005, Annual review of biomedical engineering.

[20]  M. Ferrarin,et al.  Standing-up exerciser based on functional electrical stimulation and body weight relief , 2002, Medical and Biological Engineering and Computing.

[21]  G P Braz,et al.  Electrically-evoked control of the swinging leg after spinal cord injury: open-loop or motion sensor-assisted control? , 2007, Australasian physical & engineering sciences in medicine.

[22]  Bogert Aj.  A Proportional Derivative FES Controller for Planar Arm Movement , 2007 .

[23]  L. Sheffler,et al.  Neuromuscular electrical stimulation in neurorehabilitation , 2007, Muscle & nerve.

[24]  K. Ragnarsson.  Functional electrical stimulation after spinal cord injury: current use, therapeutic effects and future directions , 2008, Spinal Cord.

[25]  O. Sujith.  Functional electrical stimulation in neurological disorders. , 2008 .

[26]  Kathleen M. Jagodnik,et al.  Creating a Reinforcement Learning Controller for Functional Electrical Stimulation of a Human Arm. , 2008, The ... Yale Workshop on Adaptive and Learning Systems.

[27]  Antonie J. van den Bogert,et al.  A Real-Time, 3-D Musculoskeletal Model for Dynamic Simulation of Arm Movements , 2009, IEEE Transactions on Biomedical Engineering.

[28]  Richard S. Sutton,et al.  Reinforcement Learning , 1992, Handbook of Machine Learning.