Stable Responsive EMG Sequence Prediction and Adaptive Reinforcement With Temporal Convolutional Networks

Prediction of movement intentions from electromyographic (EMG) signals is typically performed with a pattern recognition approach, wherein a short dataframe of raw EMG is compressed into an instantaneous feature-encoding that is meaningful for classification. However, EMG signals are time-varying, implying that a frame-wise approach may not sufficiently incorporate temporal context into predictions, leading to erratic and unstable prediction behavior. <italic>Objective:</italic> We demonstrate that sequential prediction models and, specifically, temporal convolutional networks are able to leverage useful temporal information from EMG to achieve superior predictive performance. <italic>Methods:</italic> We compare this approach to other sequential and frame-wise models predicting 3 simultaneous hand and wrist degrees-of-freedom from 2 amputee and 13 non-amputee human subjects in a minimally constrained experiment. We also compare these models on the publicly available Ninapro and CapgMyo amputee and non-amputee datasets. <italic>Results:</italic> Temporal convolutional networks yield predictions that are more accurate and stable <inline-formula><tex-math notation="LaTeX">$(p < 0.001)$</tex-math></inline-formula> than frame-wise models, especially during inter-class transitions, with an average response delay of 4.6 ms <inline-formula><tex-math notation="LaTeX">$(p < 0.001)$</tex-math></inline-formula> and simpler feature-encoding. Their performance can be further improved with adaptive reinforcement training. <italic>Significance:</italic> Sequential models that incorporate temporal information from EMG achieve superior movement prediction performance and these models allow for novel types of interactive training. <italic>Conclusions:</italic> Addressing EMG decoding as a sequential modeling problem will lead to enhancements in the reliability, responsiveness, and movement complexity available from prosthesis control systems.

[1]  Erik J. Scheme,et al.  Selective Classification for Improved Robustness of Myoelectric Control Under Nonideal Conditions , 2011, IEEE Transactions on Biomedical Engineering.

[2]  Gregory D. Hager,et al.  Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation , 2016, ECCV.

[3]  Nitish V. Thakor,et al.  Limb Position Tolerant Pattern Recognition for Myoelectric Prosthesis Control with Adaptive Sparse Representations From Extreme Learning , 2018, IEEE Transactions on Biomedical Engineering.

[4]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[5]  Kevin B. Englehart,et al.  A robust, real-time control scheme for multifunction myoelectric control , 2003, IEEE Transactions on Biomedical Engineering.

[6]  Dario Farina,et al.  Self-Correcting Pattern Recognition System of Surface EMG Signals for Upper Limb Prosthesis Control , 2014, IEEE Transactions on Biomedical Engineering.

[7]  Gregory D. Hager,et al.  Learning convolutional action primitives for fine-grained action recognition , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[8]  Vladlen Koltun,et al.  An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling , 2018, ArXiv.

[9]  Si Wu,et al.  Improving support vector machine classifiers by modifying kernel functions , 1999, Neural Networks.

[10]  Toshio Tsuji,et al.  Pattern classification of time-series EMG signals using neural networks , 2000 .

[11]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[12]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[13]  Austin Reiter,et al.  Interpretable 3D Human Action Analysis with Temporal Convolutional Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[14]  Todd A. Kuiken,et al.  A Decision-Based Velocity Ramp for Minimizing the Effect of Misclassifications During Real-Time Pattern Recognition Control , 2011, IEEE Transactions on Biomedical Engineering.

[15]  Manfredo Atzori,et al.  Comparison of six electromyography acquisition setups on hand movement classification tasks , 2017, PloS one.

[16]  Manfredo Atzori,et al.  Deep Learning with Convolutional Neural Networks Applied to Electromyography Data: A Resource for the Classification of Movements for Prosthetic Hands , 2016, Front. Neurorobot..

[17]  Blair A. Lock,et al.  Determining the Optimal Window Length for Pattern Recognition-Based Myoelectric Control: Balancing the Competing Effects of Classification Error and Controller Delay , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[18]  Kapil D. Katyal,et al.  Revolutionizing Prosthetics—Phase 3 , 2013 .

[19]  Alex Bateman,et al.  An introduction to hidden Markov models. , 2007, Current protocols in bioinformatics.

[20]  Hong Liu,et al.  Dynamic Hand Motion Recognition Based on Transient and steady-State EMG signals , 2012, Int. J. Humanoid Robotics.

[21]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[22]  Xinjun Sheng,et al.  Cascaded Adaptation Framework for Fast Calibration of Myoelectric Control , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[23]  Adrian D. C. Chan,et al.  Continuous myoelectric control for powered prostheses using hidden Markov models , 2005, IEEE Transactions on Biomedical Engineering.

[24]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[25]  R.F. Weir,et al.  The Optimal Controller Delay for Myoelectric Prostheses , 2007, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[26]  Dario Farina,et al.  Influence of the training set on the accuracy of surface EMG classification in dynamic contractions for the control of multifunction prostheses , 2011, Journal of NeuroEngineering and Rehabilitation.

[27]  Erik J. Scheme,et al.  Effects of Confidence-Based Rejection on Usability and Error in Pattern Recognition-Based Myoelectric Control , 2019, IEEE Journal of Biomedical and Health Informatics.

[28]  Ernest Nlandu Kamavuako,et al.  EVALUATION OF CLASSIFIERS PERFORMANCE USING THE MYO ARMBAND , 2017 .

[29]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Weidong Geng,et al.  Gesture recognition by instantaneous surface EMG images , 2016, Scientific Reports.

[31]  Lee E. Miller,et al.  Neural Manifolds for the Control of Movement , 2017, Neuron.

[32]  Nitish V. Thakor,et al.  Stable Electromyographic Sequence Prediction During Movement Transitions using Temporal Convolutional Networks , 2019, 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

[33]  Max Ortiz-Catalan,et al.  Offline accuracy: A potentially misleading metric in myoelectric pattern recognition for prosthetic control , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[34]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[35]  Christian Cipriani,et al.  Classification of Transient Myoelectric Signals for the Control of Multi-Grasp Hand Prostheses , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[36]  R. Duncan Luce,et al.  Individual Choice Behavior: A Theoretical Analysis , 1979 .

[37]  Erik J. Scheme,et al.  Confidence-Based Rejection for Improved Pattern Recognition Myoelectric Control , 2013, IEEE Transactions on Biomedical Engineering.

[38]  Manfredo Atzori,et al.  Electromyography data for non-invasive naturally-controlled robotic hand prostheses , 2014, Scientific Data.

[39]  Yinghong Peng,et al.  EMG‐Based Estimation of Limb Movement Using Deep Learning With Recurrent Convolutional Neural Networks , 2018, Artificial organs.

[40]  R.N. Scott,et al.  A new strategy for multifunction myoelectric control , 1993, IEEE Transactions on Biomedical Engineering.

[41]  Gregory D. Hager,et al.  Temporal Convolutional Networks for Action Segmentation and Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Richard Hans Robert Hahnloser,et al.  Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit , 2000, Nature.

[43]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[44]  W. Kruskal,et al.  Use of Ranks in One-Criterion Variance Analysis , 1952 .