Sensor Fusion for Predictive Control of Human-Prosthesis-Environment Dynamics in Assistive Walking: A Survey

This survey paper concerns Sensor Fusion for Predictive Control of Human-Prosthesis-Environment Dynamics in Assistive Walking. The powered lower limb prosthesis can imitate the human limb motion and help amputees to recover the walking ability, but it is still a challenge for amputees to walk in complex environments with the powered prosthesis. Previous researchers mainly focused on the interaction between a human and the prosthesis without considering the environmental information, which can provide an environmental context for human-prosthesis interaction. Therefore, in this review, recent sensor fusion methods for the predictive control of human-prosthesis-environment dynamics in assistive walking are critically surveyed. In that backdrop, several pertinent research issues that need further investigation are presented. In particular, general controllers, comparison of sensors, and complete procedures of sensor fusion methods that are applicable in assistive walking are introduced. Also, possible sensor fusion research for human-prosthesis-environment dynamics is presented.

[1]  K. Walas Terrain Classification and Negotiation with a Walking Robot , 2015, J. Intell. Robotic Syst..

[2]  Orhan Er,et al.  Comparison of Different Time and Frequency Domain Feature Extraction Methods on Elbow Gesture’s EMG , 2016 .

[3]  Enhao Zheng,et al.  Noncontact Capacitive Sensing-Based Locomotion Transition Recognition for Amputees With Robotic Transtibial Prostheses , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[4]  Michael Goldfarb,et al.  A Phase Variable Approach for IMU-Based Locomotion Activity Recognition , 2018, IEEE Transactions on Biomedical Engineering.

[5]  Huseyin Atakan Varol,et al.  User-Independent Intent Recognition for Lower Limb Prostheses Using Depth Sensing , 2018, IEEE Transactions on Biomedical Engineering.

[6]  Wen Zhang,et al.  Environmental Features Recognition for Lower Limb Prostheses Toward Predictive Walking , 2019, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[7]  Robert Riener,et al.  Complementary limb motion estimation for the control of active knee prostheses , 2011, Biomedizinische Technik. Biomedical engineering.

[8]  Elissa D. Ledoux Inertial Sensing for Gait Event Detection and Transfemoral Prosthesis Control Strategy , 2018, IEEE Transactions on Biomedical Engineering.

[9]  Gary Burnett,et al.  Translational Motion Tracking of Leg Joints for Enhanced Prediction of Walking Tasks , 2018, IEEE Transactions on Biomedical Engineering.

[10]  Tao Liu,et al.  Wearable Sensor System for Detecting Gait Parameters of Abnormal Gaits: A Feasibility Study , 2018, IEEE Sensors Journal.

[11]  M. Nash,et al.  Energy expenditure of trans-tibial amputees during ambulation at self-selected pace , 1994, Prosthetics and orthotics international.

[12]  Uriel Martinez-Hernandez,et al.  Adaptive Bayesian inference system for recognition of walking activities and prediction of gait events using wearable sensors , 2018, Neural Networks.

[13]  Shriya S Srinivasan,et al.  Proprioception from a neurally controlled lower-extremity prosthesis , 2018, Science Translational Medicine.

[14]  Leonidas J. Guibas,et al.  Frustum PointNets for 3D Object Detection from RGB-D Data , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[15]  Robert Riener,et al.  A survey of sensor fusion methods in wearable robotics , 2015, Robotics Auton. Syst..

[16]  Deepak Joshi,et al.  Terrain and Direction Classification of Locomotion Transitions Using Neuromuscular and Mechanical Input , 2015, Annals of Biomedical Engineering.

[17]  Xiangyun Hu,et al.  Deep fusion of multi-view and multimodal representation of ALS point cloud for 3D terrain scene recognition , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[18]  Michael Goldfarb,et al.  Multiclass Real-Time Intent Recognition of a Powered Lower Limb Prosthesis , 2010, IEEE Transactions on Biomedical Engineering.

[19]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Julius Hannink,et al.  Sensor-Based Gait Parameter Extraction With Deep Convolutional Neural Networks , 2016, IEEE Journal of Biomedical and Health Informatics.

[21]  Sebastian Scherer,et al.  VoxNet: A 3D Convolutional Neural Network for real-time object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[22]  Eric Foxlin,et al.  Pedestrian tracking with shoe-mounted inertial sensors , 2005, IEEE Computer Graphics and Applications.

[23]  Michael Goldfarb,et al.  Volitional Control of a Prosthetic Knee Using Surface Electromyography , 2011, IEEE Transactions on Biomedical Engineering.

[24]  Michael Goldfarb,et al.  Design and Control of a Powered Transfemoral Prosthesis , 2008, Int. J. Robotics Res..

[25]  Clarence W. de Silva,et al.  Sensor Systems: Fundamentals and Applications , 2016 .

[26]  Stephanie Huang,et al.  Voluntary Control of Residual Antagonistic Muscles in Transtibial Amputees: Feedforward Ballistic Contractions and Implications for Direct Neural Control of Powered Lower Limb Prostheses , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[27]  Mary M. Hayhoe,et al.  Gaze and the Control of Foot Placement When Walking in Natural Terrain , 2018, Current Biology.

[28]  Levi J. Hargrove,et al.  Depth Sensing for Improved Control of Lower Limb Prostheses , 2015, IEEE Transactions on Biomedical Engineering.

[29]  Li Ning Sun,et al.  An Echo-Based Gait Phase Determination Method of Lower Limb Prosthesis , 2013 .

[30]  Michael Goldfarb,et al.  Upslope Walking With a Powered Knee and Ankle Prosthesis: Initial Results With an Amputee Subject , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[31]  H. van der Kooij,et al.  Reference Trajectory Generation for Rehabilitation Robots: Complementary Limb Motion Estimation , 2009, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[32]  Robert Riener,et al.  Control strategies for active lower extremity prosthetics and orthotics: a review , 2015, Journal of NeuroEngineering and Rehabilitation.

[33]  Yasuhisa Hasegawa,et al.  Intention-based walking support for paraplegia patients with Robot Suit HAL , 2007, Adv. Robotics.

[34]  Omid Dehzangi,et al.  IMU-Based Gait Recognition Using Convolutional Neural Networks and Multi-Sensor Fusion , 2017, Sensors.

[35]  Rong Song,et al.  Movement Performance of Human–Robot Cooperation Control Based on EMG-Driven Hill-Type and Proportional Models for an Ankle Power-Assist Exoskeleton Robot , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[36]  Dario Farina,et al.  Online mapping of EMG signals into kinematics by autoencoding , 2018, Journal of NeuroEngineering and Rehabilitation.

[37]  Levi J. Hargrove,et al.  Fusion of Bilateral Lower-Limb Neuromechanical Signals Improves Prediction of Locomotor Activities , 2018, Front. Robot. AI.

[38]  Xuan Zhang,et al.  Human lower extremity joint moment prediction: A wavelet neural network approach , 2014, Expert Syst. Appl..

[39]  Hugh M. Herr,et al.  Powered ankle-foot prosthesis , 2008, IEEE Robotics & Automation Magazine.

[40]  Qining Wang,et al.  Real-Time On-Board Recognition of Continuous Locomotion Modes for Amputees With Robotic Transtibial Prostheses , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[41]  Eijiro Takeuchi,et al.  Autonomous predictive driving for blind intersections , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[42]  Daniel P. Ferris,et al.  Locomotor Adaptation by Transtibial Amputees Walking With an Experimental Powered Prosthesis Under Continuous Myoelectric Control , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[43]  Kathryn Ziegler-Graham,et al.  Estimating the prevalence of limb loss in the United States: 2005 to 2050. , 2008, Archives of physical medicine and rehabilitation.

[44]  Janice J Eng,et al.  Incidence of lower limb amputation in Canada , 2017, Canadian Journal of Public Health.

[45]  Hugh M. Herr,et al.  Powered Ankle--Foot Prosthesis Improves Walking Metabolic Economy , 2009, IEEE Transactions on Robotics.

[46]  Xiaohui Peng,et al.  Deep Learning for Sensor-based Activity Recognition: A Survey , 2017, Pattern Recognit. Lett..

[47]  G. Dumanian,et al.  Advances in Transfemoral Amputee Rehabilitation: Early Experience with Targeted Muscle Reinnervation , 2014, Current Surgery Reports.

[48]  Long Wang,et al.  Walk the Walk: A Lightweight Active Transtibial Prosthesis , 2015, IEEE Robotics & Automation Magazine.

[49]  Panagiotis K. Artemiadis,et al.  An EMG-Based Robot Control Scheme Robust to Time-Varying EMG Signal Features , 2010, IEEE Transactions on Information Technology in Biomedicine.

[50]  Kimberly A. Ingraham,et al.  Configuring a Powered Knee and Ankle Prosthesis for Transfemoral Amputees within Five Specific Ambulation Modes , 2014, PloS one.

[51]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[53]  Fan Zhang,et al.  Continuous Locomotion-Mode Identification for Prosthetic Legs Based on Neuromuscular–Mechanical Fusion , 2011, IEEE Transactions on Biomedical Engineering.

[54]  He Huang,et al.  A Strategy for Identifying Locomotion Modes Using Surface Electromyography , 2009, IEEE Transactions on Biomedical Engineering.

[55]  Chenglong Fu,et al.  Directional PointNet: 3D Environmental Classification for Wearable Robotics , 2019, ArXiv.

[56]  Clarence W. de Silva,et al.  Sensors and Actuators: Engineering System Instrumentation, Second Edition , 2015 .

[57]  M. Goldfarb,et al.  Control of Stair Ascent and Descent With a Powered Transfemoral Prosthesis , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[58]  Ming Liu,et al.  Development of an Environment-Aware Locomotion Mode Recognition System for Powered Lower Limb Prostheses , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[59]  Rong Song,et al.  Admittance Control Based on EMG-Driven Musculoskeletal Model Improves the Human–Robot Synchronization , 2019, IEEE Transactions on Industrial Informatics.

[60]  Mazharul Islam,et al.  Detection of Gait Modes Using an Artificial Neural Network during Walking with a Powered Ankle-Foot Orthosis , 2016, Journal of biophysics.