Neurorobotic fusion of prosthetic touch, kinesthesia, and movement in bionic upper limbs promotes intrinsic brain behaviors

Description Multiperspective analysis reveals neurorobotic sensory and motor fusion in a bionic system promotes intrinsic neural behaviors. Bionic prostheses have restorative potential. However, the complex interplay between intuitive motor control, proprioception, and touch that represents the hallmark of human upper limb function has not been revealed. Here, we show that the neurorobotic fusion of touch, grip kinesthesia, and intuitive motor control promotes levels of behavioral performance that are stratified toward able-bodied function and away from standard-of-care prosthetic users. This was achieved through targeted motor and sensory reinnervation, a closed-loop neural-machine interface, coupled to a noninvasive robotic architecture. Adding touch to motor control improves the ability to reach intended target grasp forces, find target durometers among distractors, and promote prosthetic ownership. Touch, kinesthesia, and motor control restore balanced decision strategies when identifying target durometers and intrinsic visuomotor behaviors that reduce the need to watch the prosthetic hand during object interactions, which frees the eyes to look ahead to the next planned action. The combination of these three modalities also enhances error correction performance. We applied our unified theoretical, functional, and clinical analyses, enabling us to define the relative contributions of the sensory and motor modalities operating simultaneously in this neural-machine interface. This multiperspective framework provides the necessary evidence to show that bionic prostheses attain more human-like function with effective sensory-motor restoration.

[1]  Stephanie L Carey,et al.  Compensatory movements of transradial prosthesis users during common tasks. , 2008, Clinical biomechanics.

[2]  Shriya S. Srinivasan,et al.  Neural interfacing architecture enables enhanced motor control and residual limb functionality postamputation , 2021, Proceedings of the National Academy of Sciences.

[3]  Christian Cipriani,et al.  Improving internal model strength and performance of prosthetic hands using augmented feedback , 2018, Journal of NeuroEngineering and Rehabilitation.

[4]  Antonio Frisoli,et al.  Continuous supplementary tactile feedback can be applied (and then removed) to enhance precision manipulation , 2020, Journal of neuroengineering and rehabilitation.

[5]  Mai Dwairy,et al.  The application of foraging theory to the information searching behaviour of general practitioners , 2011, BMC family practice.

[6]  Daniel H. Blustein,et al.  Crossmodal congruency effect scores decrease with repeat test exposure , 2019, PeerJ.

[7]  Scott M Tintle,et al.  Targeted Muscle Reinnervation Treats Neuroma and Phantom Pain in Major Limb Amputees: A Randomized Clinical Trial. , 2019, Annals of surgery.

[8]  Michael L. Boninger,et al.  Implicit Grasp Force Representation in Human Motor Cortical Recordings , 2018, Front. Neurosci..

[9]  L Kenney,et al.  Upper limb activity in myoelectric prosthesis users is biased towards the intact limb and appears unrelated to goal-directed task performance , 2018, Scientific Reports.

[10]  Nathan T. Kearns,et al.  Evaluation of Performance‐Based Outcome Measures for the Upper Limb: A Comprehensive Narrative Review , 2018, PM & R : the journal of injury, function, and rehabilitation.

[11]  Ahmed W Shehata,et al.  Audible Feedback Improves Internal Model Strength and Performance of Myoelectric Prosthesis Control , 2018, Scientific Reports.

[12]  Albert H Vette,et al.  Cluster-based upper body marker models for three-dimensional kinematic analysis: Comparison with an anatomical model and reliability analysis. , 2018, Journal of biomechanics.

[13]  T. Makin,et al.  Neurocognitive barriers to the embodiment of technology , 2017, Nature Biomedical Engineering.

[14]  F. Clippinger,et al.  A sensory feedback system for an upper-limb amputation prosthesis. , 1974, Bulletin of prosthetics research.

[15]  Craig S. Chapman,et al.  Quantitative Eye Gaze and Movement Differences in Visuomotor Adaptations to Varying Task Demands Among Upper-Extremity Prosthesis Users , 2019, JAMA network open.

[16]  S. Micera,et al.  Toward higher-performance bionic limbs for wider clinical use , 2021, Nature Biomedical Engineering.

[17]  Rafael Granja-Vazquez,et al.  Illusory movement perception improves motor control for prosthetic hands , 2018, Science Translational Medicine.

[18]  Matthew Fielding,et al.  ‘Disrupting the Optimal Forager’: Predictive Risk Mapping and Domestic Burglary Reduction in Trafford, Greater Manchester , 2012 .

[19]  Shriya S. Srinivasan,et al.  Caprine Models of the Agonist-Antagonist Myoneural Interface Implemented at the Above- and Below-Knee Amputation Levels. , 2019, Plastic and reconstructive surgery.

[20]  Christian Cipriani,et al.  Neural feedback strategies to improve grasping coordination in neuromusculoskeletal prostheses , 2020, Scientific Reports.

[21]  Albert H Vette,et al.  Using synchronized eye and motion tracking to determine high-precision eye-movement patterns during object-interaction tasks. , 2018, Journal of vision.

[22]  Albert H Vette,et al.  Use of optical motion capture for the analysis of normative upper body kinematics during functional upper limb tasks: A systematic review. , 2018, Journal of electromyography and kinesiology : official journal of the International Society of Electrophysiological Kinesiology.

[23]  T. Kuiken,et al.  Sensory capacity of reinnervated skin after redirection of amputated upper limb nerves to the chest , 2009, Brain : a journal of neurology.

[24]  Z. C. Thumser,et al.  Reliability in evaluator-based tests: using simulation-constructed models to determine contextually relevant agreement thresholds , 2018, BMC Medical Research Methodology.

[25]  D. Silcox,et al.  Myoelectric prostheses. A long-term follow-up and a study of the use of alternate prostheses. , 1993, The Journal of bone and joint surgery. American volume.

[26]  W. B. Whalley Evaluating student assessments: the use of optimal foraging theory , 2016 .

[27]  Dario Farina,et al.  The clinical relevance of advanced artificial feedback in the control of a multi-functional myoelectric prosthesis , 2018, Journal of NeuroEngineering and Rehabilitation.

[28]  A. Eliasson,et al.  Assessment of capacity for myoelectric control: a new Rasch-built measure of prosthetic hand control. , 2005, Journal of rehabilitation medicine.

[29]  John W. Krakauer,et al.  Independent learning of internal models for kinematic and dynamic control of reaching , 1999, Nature Neuroscience.

[30]  M. Clark,et al.  Longitudinal study of prosthesis use in veterans with upper limb amputation. , 2020, Prosthetics and orthotics international.

[31]  Dario Farina,et al.  Tactile feedback is an effective instrument for the training of grasping with a prosthesis at low- and medium-force levels , 2017, Experimental Brain Research.

[32]  Keehoon Kim,et al.  Haptic Feedback Enhances Grip Force Control of sEMG-Controlled Prosthetic Hands in Targeted Reinnervation Amputees , 2012, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[33]  Mohammad M D Sobuh,et al.  Visuomotor behaviours when using a myoelectric prosthesis , 2014, Journal of NeuroEngineering and Rehabilitation.

[34]  A. Bastian Understanding sensorimotor adaptation and learning for rehabilitation , 2008, Current opinion in neurology.

[35]  Silvestro Micera,et al.  Enhancing functional abilities and cognitive integration of the lower limb prosthesis , 2019, Science Translational Medicine.

[36]  T. Kuiken,et al.  Vibrotactile detection thresholds for chest skin of amputees following targeted reinnervation surgery , 2009, Brain Research.

[37]  Marco Vilela,et al.  Applications of brain-computer interfaces to the control of robotic and prosthetic arms. , 2020, Handbook of clinical neurology.

[38]  Linda Resnik,et al.  Home Use of a Neural-connected Sensory Prosthesis Provides the Functional and Psychosocial Experience of Having a Hand Again , 2018, Scientific Reports.

[39]  Jacqueline S. Hebert,et al.  Novel Targeted Sensory Reinnervation Technique to Restore Functional Hand Sensation After Transhumeral Amputation , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[40]  Christina Rock Gambrell Overuse Syndrome and the Unilateral Upper Limb Amputee: Consequences and Prevention , 2008 .

[41]  Andrea Cimolato,et al.  Neural signal recording and processing in somatic neuroprosthetic applications. A review , 2020, Journal of Neuroscience Methods.

[42]  Patrick M. Pilarski,et al.  Gaze and Movement Assessment (GaMA): Inter-site validation of a visuomotor upper limb functional protocol , 2019, bioRxiv.

[43]  Dustin J Tyler,et al.  The benefits of sensation on the experience of a hand: A qualitative case series , 2019, PloS one.

[44]  M. Keith,et al.  A neural interface provides long-term stable natural touch perception , 2014, Science Translational Medicine.

[45]  Øyvind Stavdahl,et al.  Upper Limb Prosthetic Outcome Measures (ULPOM): A Working Group and Their Findings , 2009 .

[46]  Z. C. Thumser,et al.  Using sensory discrimination in a foraging-style task to evaluate human upper-limb sensorimotor performance , 2019, Scientific Reports.

[47]  Keehoon Kim,et al.  Robotic touch shifts perception of embodiment to a prosthesis in targeted reinnervation amputees. , 2011, Brain : a journal of neurology.

[48]  Enzo Mastinu,et al.  Self-Contained Neuromusculoskeletal Arm Prostheses. , 2020, The New England journal of medicine.

[49]  Helen Y N Lindner,et al.  Upper Limb Prosthetic Outcome Measures: Review and Content Comparison Based on International Classification of Functioning, Disability and Health , 2010, Prosthetics and orthotics international.

[50]  Thierry Keller,et al.  Short- and Long-Term Learning of Feedforward Control of a Myoelectric Prosthesis with Sensory Feedback by Amputees , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[51]  Andreas Kalckert,et al.  The Onset Time of the Ownership Sensation in the Moving Rubber Hand Illusion , 2017, Front. Psychol..

[52]  Elizaveta V Okorokova,et al.  Biomimetic sensory feedback through peripheral nerve stimulation improves dexterous use of a bionic hand , 2019, Science Robotics.

[53]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[54]  G.S. Dhillon,et al.  Direct neural sensory feedback and control of a prosthetic arm , 2005, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[55]  Albert H Vette,et al.  Gaze and Movement Assessment (GaMA): Inter-site validation of a visuomotor upper limb functional protocol , 2019, PloS one.

[56]  Craig S. Chapman,et al.  Compensatory strategies of body-powered prosthesis users reveal primary reliance on trunk motion and relation to skill level. , 2019, Clinical biomechanics.

[57]  Christian Antfolk,et al.  A review of invasive and non-invasive sensory feedback in upper limb prostheses , 2017, Expert review of medical devices.

[58]  Silvestro Micera,et al.  A closed-loop hand prosthesis with simultaneous intraneural tactile and position feedback , 2018, Science Robotics.

[59]  G. Wood,et al.  Examining the Spatiotemporal Disruption to Gaze When Using a Myoelectric Prosthetic Hand , 2018, Journal of motor behavior.

[60]  Silvestro Micera,et al.  Intraneural sensory feedback restores grip force control and motor coordination while using a prosthetic hand , 2019, Journal of neural engineering.

[61]  Pratik K. Mutha,et al.  The influence of visual target information on the online control of movements , 2015, Vision Research.

[62]  Z. C. Thumser,et al.  Long-Term Home-Use of Sensory-Motor-Integrated Bidirectional Bionic Prosthetic Arms Promotes Functional, Perceptual, and Cognitive Changes , 2020, Frontiers in Neuroscience.

[63]  Linda Resnik,et al.  Systematic Review of Measures of Impairment and Activity Limitation for Persons With Upper Limb Trauma and Amputation. , 2017, Archives of physical medicine and rehabilitation.

[64]  Jacqueline S. Hebert,et al.  Updates in Targeted Sensory Reinnervation for Upper Limb Amputation , 2014, Current Surgery Reports.

[65]  F. Mussa-Ivaldi,et al.  The motor system does not learn the dynamics of the arm by rote memorization of past experience. , 1997, Journal of neurophysiology.

[66]  Silvestro Micera,et al.  Multisensory bionic limb to achieve prosthesis embodiment and reduce distorted phantom limb perceptions , 2018, Journal of Neurology, Neurosurgery, and Psychiatry.

[67]  Konrad P. Körding,et al.  Uncertainty of Feedback and State Estimation Determines the Speed of Motor Adaptation , 2009, Front. Comput. Neurosci..

[68]  Albert H Vette,et al.  Characterization of normative hand movements during two functional upper limb tasks , 2018, PloS one.

[69]  Dustin J. Tyler,et al.  Learning of Artificial Sensation Through Long-Term Home Use of a Sensory-Enabled Prosthesis , 2019, Front. Neurosci..

[70]  Christian Cipriani,et al.  Humans can integrate feedback of discrete events in their sensorimotor control of a robotic hand , 2014, Experimental Brain Research.

[71]  Silvestro Micera,et al.  Biomimetic Intraneural Sensory Feedback Enhances Sensation Naturalness, Tactile Sensitivity, and Manual Dexterity in a Bidirectional Prosthesis , 2018, Neuron.

[72]  Stefania Fatone,et al.  Comparison of range-of-motion and variability in upper body movements between transradial prosthesis users and able-bodied controls when executing goal-oriented tasks , 2014, Journal of NeuroEngineering and Rehabilitation.

[73]  Dario Farina,et al.  The Extraction of Neural Information from the Surface EMG for the Control of Upper-Limb Prostheses: Emerging Avenues and Challenges , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[74]  Olaf Blanke,et al.  Behavioral, Neural, and Computational Principles of Bodily Self-Consciousness , 2015, Neuron.

[75]  Gregory A. Clark,et al.  Motor Control and Sensory Feedback Enhance Prosthesis Embodiment and Reduce Phantom Pain After Long-Term Hand Amputation , 2018, Front. Hum. Neurosci..

[76]  Jonathon W. Sensinger,et al.  A Review of Sensory Feedback in Upper-Limb Prostheses From the Perspective of Human Motor Control , 2020, Frontiers in Neuroscience.

[77]  Albert H Vette,et al.  Characterization of normative angular joint kinematics during two functional upper limb tasks. , 2019, Gait & posture.

[78]  David J. Warren,et al.  Restoration of motor control and proprioceptive and cutaneous sensation in humans with prior upper-limb amputation via multiple Utah Slanted Electrode Arrays (USEAs) implanted in residual peripheral arm nerves , 2017, Journal of NeuroEngineering and Rehabilitation.

[79]  L. Resnik,et al.  Development and evaluation of the activities measure for upper limb amputees. , 2013, Archives of Physical Medicine and Rehabilitation.

[80]  K. Kaufman,et al.  Interfaces with the peripheral nervous system for the control of a neuroprosthetic limb: a review , 2020, Journal of NeuroEngineering and Rehabilitation.

[81]  Blair A. Lock,et al.  Redirection of cutaneous sensation from the hand to the chest skin of human amputees with targeted reinnervation , 2007, Proceedings of the National Academy of Sciences.

[82]  Adam Wilson,et al.  Assessing the quality of supplementary sensory feedback using the crossmodal congruency task , 2017, Scientific Reports.

[83]  P. Sandstrom Scholars as Subsistence Foragers , 2005 .

[84]  Dustin J Tyler,et al.  Artificial tactile and proprioceptive feedback improves performance and confidence on object identification tasks , 2018, PloS one.

[85]  J. Lackner,et al.  Rapid adaptation to Coriolis force perturbations of arm trajectory. , 1994, Journal of neurophysiology.

[86]  Ahmed W. Shehata,et al.  Evaluating Internal Model Strength and Performance of Myoelectric Prosthesis Control Strategies , 2017, bioRxiv.

[87]  K. Horch,et al.  Object Discrimination With an Artificial Hand Using Electrical Stimulation of Peripheral Tactile and Proprioceptive Pathways With Intrafascicular Electrodes , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[88]  Robert D. Lipschutz,et al.  Targeted reinnervation for enhanced prosthetic arm function in a woman with a proximal amputation: a case study , 2007, The Lancet.

[89]  Kevin Englehart,et al.  Conventional analysis of trial-by-trial adaptation is biased: Empirical and theoretical support using a Bayesian estimator , 2018, PLoS Comput. Biol..

[90]  Adam Wilson,et al.  The Crossmodal Congruency Effect, a Tool Incorporation Metric, Suffers from a Learning Effect with Repeated Exposures , 2017, bioRxiv.

[91]  Jacob L. Segil,et al.  Combination of Simultaneous Artificial Sensory Percepts to Identify Prosthetic Hand Postures: A Case Study , 2020, Scientific Reports.

[92]  Todd A Kuiken,et al.  Targeted Muscle Reinnervation for the Upper and Lower Extremity , 2017, Techniques in orthopaedics.

[93]  Z. C. Thumser,et al.  Fitts’ Law in the Control of Isometric Grip Force With Naturalistic Targets , 2018, Front. Psychol..

[94]  Anders Skrondal,et al.  Musculoskeletal pain and overuse syndromes in adult acquired major upper-limb amputees. , 2011, Archives of physical medicine and rehabilitation.

[95]  Aaron M. Dollar,et al.  Analyzing at-home prosthesis use in unilateral upper-limb amputees to inform treatment & device design , 2017, 2017 International Conference on Rehabilitation Robotics (ICORR).

[96]  Silvestro Micera,et al.  Restoration of sensory information via bionic hands , 2020, Nature Biomedical Engineering.

[97]  Jacqueline S. Hebert,et al.  Cutaneous sensory outcomes from three transhumeral targeted reinnervation cases , 2016, Prosthetics and orthotics international.