An Attention-Based CNN-LSTM Model with Limb Synergy for Joint Angles Prediction*

Estimation of lower limb movement is crucial in exoskeleton-assisted gait rehabilitation which can reduce the training load by recognizing the movement intention of patients, so as to realize the adaptive and transparent robotic assistance. Human locomotion has inherent synergies and coordination, and the dynamic mapping of the upper and lower limbs is beneficial to improve the prediction accuracy. Current prediction methods do not fully consider the correlation of gait data in time and space, resulting in a large amount of redundant data and low prediction accuracy. This paper proposes a gait trajectory prediction method based on attention-based CNN-LSTM model, which predicts the human knee/ankle joint trajectory based on upper and lower limb collaborative data. The attention mechanism is applied to determine which dimensions are essential in estimation of lower limb movement, so the accuracy can be improved by adopting key elements. Results show that, within a predicted horizon of 60 ms, prediction RMSE is as low as 0.317 degrees.

[1]  V. Dietz,et al.  Arm to leg coordination in humans during walking, creeping and swimming activities , 2001, Experimental Brain Research.

[2]  A. Lamontagne,et al.  The coordination of upper and lower limb movements during gait in healthy and stroke individuals. , 2009, Gait & posture.

[3]  Kurt L. Mudie,et al.  Lower Limb Kinematics Trajectory Prediction Using Long Short-Term Memory Neural Networks , 2020, Frontiers in Bioengineering and Biotechnology.

[4]  Arvind Gautam,et al.  MyoNet: A Transfer-Learning-Based LRCN for Lower Limb Movement Recognition and Knee Joint Angle Prediction for Remote Monitoring of Rehabilitation Progress From sEMG , 2020, IEEE Journal of Translational Engineering in Health and Medicine.

[5]  Wei Qi Yan,et al.  Gait recognition using multichannel convolution neural networks , 2019, Neural Computing and Applications.

[6]  Khalil Alipour,et al.  Synergy-based Gaussian Process Estimation of Ankle Angle and Torque: Conceptualization for High level Controlling of Active Foot Prostheses/Orthoses. , 2018, Journal of biomechanical engineering.

[7]  Ian R. Manchester,et al.  Human locomotion analysis: Identifying a dynamic mapping between upper and lower limb joints using the Koopman operator , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[8]  Yudong Zhang,et al.  Intelligent Prediction of Human Lower Extremity Joint Moment: An Artificial Neural Network Approach , 2019, IEEE Access.

[9]  Guo Dan,et al.  An LSTM-Based Prediction Method for Lower Limb Intention Perception by Integrative Analysis of Kinect Visual Signal , 2020, Journal of healthcare engineering.

[10]  U. Jakobsson.  Ageing and health , 2010, The Lancet.

[11]  Chengyu Liu,et al.  An Attention Based CNN-LSTM Approach for Sleep-Wake Detection With Heterogeneous Sensors , 2020, IEEE journal of biomedical and health informatics.

[12]  B. Gialanella,et al.  Prediction of functional outcomes in stroke patients: the role of motor patterns according to limb synergies , 2015, Aging Clinical and Experimental Research.

[13]  Ajmal Mian,et al.  Predicting Athlete Ground Reaction Forces and Moments From Spatio-Temporal Driven CNN Models , 2019, IEEE Transactions on Biomedical Engineering.

[14]  Farzam Farahmand,et al.  Coordinated activities of trunk and upper extremity muscles during walker-assisted paraplegic gait: A synergy study. , 2018, Human movement science.

[15]  Tingfang Yan,et al.  Review of assistive strategies in powered lower-limb orthoses and exoskeletons , 2015, Robotics Auton. Syst..

[16]  Chunjie Chen,et al.  Deep Spatial-Temporal Model for rehabilitation gait: optimal trajectory generation for knee joint of lower-limb exoskeleton , 2017 .

[17]  Carlo Menon,et al.  Estimating Lower Extremity Running Gait Kinematics with a Single Accelerometer: A Deep Learning Approach , 2020, Sensors.

[18]  Shen Su,et al.  Smart healthcare-oriented online prediction of lower-limb kinematics and kinetics based on data-driven neural signal decoding , 2021, Future Gener. Comput. Syst..

[19]  Peng Yuan,et al.  Walking Gait Phase Detection Based on Acceleration Signals Using LSTM-DNN Algorithm , 2019, Algorithms.

[20]  Jing Gao,et al.  Abnormal Gait Recognition Algorithm Based on LSTM-CNN Fusion Network , 2019, IEEE Access.

[21]  Zhao Ren,et al.  An attention‐based CNN‐LSTM‐BiLSTM model for short‐term electric load forecasting in integrated energy system , 2020 .

[22]  中国残疾人联合会.  中国残疾人事业统计年鉴 = China statistical yearbook on the work for persons with disabilities , 2014 .

[23]  Jingang Yi,et al.  Real-Time Intended Knee Joint Motion Prediction by Deep-Recurrent Neural Networks , 2019, IEEE Sensors Journal.

[24]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[25]  Fei Li,et al.  Prediction of Limb Joint Angles Based on Multi-Source Signals by GS-GRNN for Exoskeleton Wearer , 2020, Sensors.