Pavlovian control of intraspinal microstimulation to produce over-ground walking

Objective Neural interface technologies are more commonly used in people with neural injury. To achieve a symbiotic relationship between device and user, the control system of the device must augment remaining function and adapt to day-to-day changes. The goal of this study was to develop predictive control strategies to produce alternating, over-ground walking in a cat model of hemisection spinal cord injury (SCI) using intraspinal microstimulation (ISMS). Approach Eight cats were anaesthetized and placed in a sling over a walkway. The residual function of a hemisection SCI was mimicked by manually moving one hind-limb through the walking cycle over the walkway. ISMS targeted motor networks in the lumbosacral enlargement to activate muscles in the other limb using low levels of current (< 130 µA). Four different people took turns to move the “intact” limb. Two control strategies, which used ground reaction force and angular velocity information about the manually moved limb to control the timing of the transitions of the other limb, were compared. The first strategy, reaction-based control, used thresholds on the sensor values to initiate state transitions. The other strategy used a reinforcement learning strategy, Pavlovian control, to learn predictions about the sensor values. Thresholds on the predictions were used to initiate transitions. Main Results Both control strategies were able to produce alternating, over-ground walking. Reaction-based control required manual tuning of the thresholds for each person to produce walking, whereas Pavlovian control did not. We demonstrate that learning occurs quickly during walking. Predictions of the sensor signals were learned quickly, initiating transitions in no more than 4 steps. Pavlovian control was resilient to transitions between people walking the limb, between cat experiments, and recovered from mistakes during walking. Significance This work demonstrates that Pavlovian control can augment remaining function and allow for personalized walking with minimal tuning requirements.

[1]  I. Pavlov,et al.  The Work of the Digestive Glands , 1903, Bristol Medico-Chirurgical Journal (1883).

[2]  T. Brown On the nature of the fundamental activity of the nervous centres; together with an analysis of the conditioning of rhythmic activity in progression, and a theory of the evolution of function in the nervous system , 1914, The Journal of physiology.

[3]  I. Engberg,et al.  An electromyographic analysis of muscular activity in the hindlimb of the cat during unrestrained locomotion. , 1969, Acta physiologica Scandinavica.

[4]  G. E. Goslow,et al.  The cat step cycle: Hind limb joint angles and muscle lengths during unrestrained locomotion , 1973, Journal of morphology.

[5]  J P Paul,et al.  Hybrid FES orthosis incorporating closed loop control and sensory feedback. , 1988, Journal of biomedical engineering.

[6]  C. A. Kirkwood,et al.  Finite state control of FES systems: application of AI inductive learning techniques , 1989, Images of the Twenty-First Century. Proceedings of the Annual International Engineering in Medicine and Biology Society,.

[7]  C. A. Kirkwood,et al.  Automatic detection of gait events: a case study using inductive learning techniques. , 1989, Journal of biomedical engineering.

[8]  E. Roth,et al.  Traumatic cervical Brown-Sequard and Brown-Sequard-plus syndromes: the spectrum of presentations and outcomes , 1991, Paraplegia.

[9]  William W. Armstrong,et al.  Evaluation of Adaptive Logic Networks for control of walking in paralyzed patients , 1992, 1992 14th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[10]  D B Popović,et al.  Finite state model of locomotion for functional electrical stimulation systems. , 1993, Progress in brain research.

[11]  J. Abbas,et al.  Experimental evaluation of an adaptive feedforward controller for use in functional neuromuscular stimulation systems , 1993, Proceedings of the 15th Annual International Conference of the IEEE Engineering in Medicine and Biology Societ.

[12]  J. Hunter,et al.  Segmental effects of epidural spinal cord stimulation in humans. , 1994, The Journal of physiology.

[13]  D.B. Popovic,et al.  Machine learning in control of functional electrical stimulation systems for locomotion , 1995, IEEE Transactions on Biomedical Engineering.

[14]  E. Chaplin Functional neuromuscular stimulation for mobility in people with spinal cord injuries. The Parastep I System. , 1996, The journal of spinal cord medicine.

[15]  J J Abbas,et al.  Experimental evaluation of an adaptive feedforward controller for use in functional neuromuscular stimulation systems. , 1993, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[16]  E. Zehr,et al.  Cutaneous reflexes during human gait: electromyographic and kinematic responses to electrical stimulation. , 1997, Journal of neurophysiology.

[17]  V. Vanderhorst,et al.  Organization of lumbosacral motoneuronal cell groups innervating hindlimb, pelvic floor, and axial muscles in the cat , 1997, The Journal of comparative neurology.

[18]  M H Granat,et al.  Two artificial neural systems for generation of gait swing by means of neuromuscular electrical stimulation. , 1997, Medical engineering & physics.

[19]  R J Triolo,et al.  Muscle selection and walking performance of multichannel FES systems for ambulation in paraplegia. , 1997, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[20]  V. Mushahwar,et al.  Selective Activation and Graded Recruitment of Functional Muscle Groups through Spinal Cord Stimulation , 1998, Annals of the New York Academy of Sciences.

[21]  E. Zehr,et al.  What functions do reflexes serve during human locomotion? , 1999, Progress in Neurobiology.

[22]  R B Stein,et al.  Optimal control of walking with functional electrical stimulation: a computer simulation study. , 1999, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[23]  D M Durand,et al.  Neurofuzzy adaptive controlling of selective stimulation for FES: a case study. , 1999, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[24]  A. Prochazka,et al.  Spinal Cord Microstimulation Generates Functional Limb Movements in Chronically Implanted Cats , 2000, Experimental Neurology.

[25]  V. Mushahwar,et al.  Selective activation of muscle groups in the feline hindlimb through electrical microstimulation of the ventral lumbo-sacral spinal cord. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[26]  R Williamson,et al.  Gait event detection for FES using accelerometers and supervised machine learning. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[27]  N Fisekovic,et al.  New controller for functional electrical stimulation systems. , 2001, Medical engineering & physics.

[28]  JOHN W. Moore A Neuroscientist's Guide to Classical Conditioning , 2002 .

[29]  E. Kehoe,et al.  Fundamental Behavioral Methods and Findings in Classical Conditioning , 2002 .

[30]  K. Pearson,et al.  Contribution of sensory feedback to ongoing ankle extensor activity during the stance phase of walking. , 2004, Canadian journal of physiology and pharmacology.

[31]  Jiping He,et al.  Epidural spinal-cord stimulation facilitates recovery of functional walking following incomplete spinal-cord injury , 2004, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[32]  V. Mushahwar,et al.  Intraspinal microstimulation generates functional movements after spinal-cord injury , 2004, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[33]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 2004, Machine Learning.

[34]  K. Anderson Targeting recovery: priorities of the spinal cord-injured population. , 2004, Journal of neurotrauma.

[35]  T. Johnston,et al.  Implantable FES system for upright mobility and bladder and bowel function for individuals with spinal cord injury , 2005, Spinal Cord.

[36]  B. Skinner Operant Behavior , 2021, Encyclopedia of Evolutionary Psychological Science.

[37]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[38]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[39]  G. M. Lyons,et al.  Finite state control of functional electrical stimulation for the rehabilitation of gait , 2000, Medical and Biological Engineering and Computing.

[40]  Duo Xu,et al.  Role of the Olivo-Cerebellar System in Timing , 2006, The Journal of Neuroscience.

[41]  K. Y. Tong,et al.  Gait control system for functional electrical stimulation using neural networks , 2006, Medical & Biological Engineering & Computing.

[42]  K. Pearson,et al.  Long-lasting, context-dependent modification of stepping in the cat after repeated stumbling-corrective responses. , 2007, Journal of neurophysiology.

[43]  V. Mushahwar,et al.  A Physiologically-based Controller for Generating Overground Locomotion using Functional Electrical Stimulation , 2007 .

[44]  V. Mushahwar,et al.  Strategies for Generating Prolonged Functional Standing Using Intramuscular Stimulation or Intraspinal Microstimulation , 2007, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[45]  R. Kobetic,et al.  Walking after incomplete spinal cord injury using an implanted FES system: a case report. , 2007, Journal of rehabilitation research and development.

[46]  P. Dean,et al.  Evidence from retractor bulbi EMG for linearized motor control of conditioned nictitating membrane responses. , 2007, Journal of neurophysiology.

[47]  E. Fetz,et al.  Direct control of paralyzed muscles by cortical neurons , 2008, Nature.

[48]  Yael Niv,et al.  Operant Conditioning , 1971 .

[49]  D. Marigold Role of Peripheral Visual Cues in Online Visual Guidance of Locomotion , 2008, Exercise and sport sciences reviews.

[50]  Jaynie F. Yang,et al.  Training of Walking Skills Overground and on the Treadmill: Case Series on Individuals With Incomplete Spinal Cord Injury , 2009, Physical Therapy.

[51]  V. Dietz,et al.  Outcome after incomplete spinal cord injury: central cord versus Brown-Sequard syndrome , 2010, Spinal Cord.

[52]  Patrick M. Pilarski,et al.  Horde: a scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction , 2011, AAMAS.

[53]  Richard B. Stein,et al.  Restoring stepping after spinal cord injury using intraspinal microstimulation and novel control strategies , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[54]  R. S. Sutton,et al.  Dynamic switching and real-time machine learning for improved human control of assistive biomedical robots , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[55]  Patrick M. Pilarski,et al.  Adaptive artificial limbs: a real-time approach to prediction and anticipation , 2013, IEEE Robotics & Automation Magazine.

[56]  Patrick M. Pilarski,et al.  Temporal-Difference Learning to Assist Human Decision Making during the Control of an Artificial Limb , 2013, ArXiv.

[57]  R B Stein,et al.  Real-time control of walking using recordings from dorsal root ganglia , 2013, Journal of neural engineering.

[58]  Patrick M. Pilarski,et al.  Real-time prediction learning for the simultaneous actuation of multiple prosthetic joints , 2013, 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR).

[59]  Richard S. Sutton,et al.  Prediction Driven Behavior: Learning Predictions that Drive Fixed Responses , 2014, AAAI 2014.

[60]  P. Pilarski,et al.  Multilayer General Value Functions for Robotic Prediction and Control , 2014 .

[61]  Richard S. Sutton,et al.  True Online TD(lambda) , 2014, ICML.

[62]  T. Lam,et al.  Training with robot-applied resistance in people with motor-incomplete spinal cord injury: Pilot study. , 2015, Journal of rehabilitation research and development.

[63]  Adam M White,et al.  DEVELOPING A PREDICTIVE APPROACH TO KNOWLEDGE , 2015 .

[64]  Roger D Quinn,et al.  Powered Lower-Limb Exoskeletons to Restore Gait for Individuals with Paraplegia - a Review. , 2015, Case orthopaedic journal.

[65]  Simon M. Danner,et al.  Augmentation of Voluntary Locomotor Activity by Transcutaneous Spinal Cord Stimulation in Motor-Incomplete Spinal Cord-Injured Individuals. , 2015, Artificial organs.

[66]  R. Stein,et al.  Intraspinal microstimulation produces over-ground walking in anesthetized cats , 2016, Journal of neural engineering.

[67]  Craig Sherstan,et al.  Application of real-time machine learning to myoelectric prosthesis control: A case series in adaptive switching , 2016, Prosthetics and orthotics international.

[68]  Spinal Cord Injury (SCI) 2016 Facts and Figures at a Glance , 2016, The journal of spinal cord medicine.

[69]  Qin,et al.  A Brain–Spinal Interface Alleviating Gait Deficits after Spinal Cord Injury in Primates , 2017 .

[70]  Patrick M. Pilarski,et al.  True Online Temporal-Difference Learning , 2015, J. Mach. Learn. Res..

[71]  SCI Facts and Figures , 2017, The journal of spinal cord medicine.

[72]  A. Dalrymple,et al.  Stimulation of the Spinal Cord for the Control of Walking , 2017 .

[73]  Patrick M. Pilarski,et al.  Representing high-dimensional data to intelligent prostheses and other wearable assistive robots: A first comparison of tile coding and selective Kanerva coding , 2017, 2017 International Conference on Rehabilitation Robotics (ICORR).

[74]  Miranda A Mathews,et al.  Reviewing the Role of the Efferent Vestibular System in Motor and Vestibular Circuits , 2017, Front. Physiol..

[75]  V. Mushahwar,et al.  The Fabrication, Implantation, and Stability of Intraspinal Microwire Arrays in the Spinal Cord of Cat and Rat , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[76]  Ashley N Dalrymple,et al.  A speed-adaptive intraspinal microstimulation controller to restore weight-bearing stepping in a spinal cord hemisection model , 2018, Journal of neural engineering.

[77]  V. Kunam,et al.  Incomplete Cord Syndromes: Clinical and Imaging Review. , 2018, Radiographics : a review publication of the Radiological Society of North America, Inc.

[78]  Irfan Essa,et al.  Identifying Gait Phases from Joint Kinematics during Walking with Switched Linear Dynamical Systems , 2018 .

[79]  Igor A. Lavrov,et al.  Neuromodulation of lumbosacral spinal networks enables independent stepping after complete paraplegia , 2018, Nature Medicine.

[80]  M. Goldfarb,et al.  Supplemental Stimulation Improves Swing Phase Kinematics During Exoskeleton Assisted Gait of SCI Subjects With Severe Muscle Spasticity , 2018, Front. Neurosci..

[81]  Fabien B. Wagner,et al.  Targeted neurotechnology restores walking in humans with spinal cord injury , 2018, Nature.

[82]  S. Morrison,et al.  Longitudinal Recovery and Reduced Costs After 120 Sessions of Locomotor Training for Motor Incomplete Spinal Cord Injury. , 2017, Archives of Physical Medicine and Rehabilitation.

[83]  Christie K. Ferreira,et al.  Recovery of Over‐Ground Walking after Chronic Motor Complete Spinal Cord Injury , 2018, The New England journal of medicine.

[84]  Gary S Bhumbra,et al.  Recurrent excitation between motoneurones propagates across segments and is purely glutamatergic , 2017, bioRxiv.

[85]  Jennie Si,et al.  Online Reinforcement Learning Control for the Personalization of a Robotic Knee Prosthesis , 2020, IEEE Transactions on Cybernetics.