Interpreting Deep Learning Features for Myoelectric Control: A Comparison With Handcrafted Features

Existing research on myoelectric control systems primarily focuses on extracting discriminative characteristics of the electromyographic (EMG) signal by designing handcrafted features. Recently, however, deep learning techniques have been applied to the challenging task of EMG-based gesture recognition. The adoption of these techniques slowly shifts the focus from feature engineering to feature learning. Nevertheless, the black-box nature of deep learning makes it hard to understand the type of information learned by the network and how it relates to handcrafted features. Additionally, due to the high variability in EMG recordings between participants, deep features tend to generalize poorly across subjects using standard training methods. Consequently, this work introduces a new multi-domain learning algorithm, named ADANN (Adaptive Domain Adversarial Neural Network), which significantly enhances (p = 0.00004) inter-subject classification accuracy by an average of 19.40% compared to standard training. Using ADANN-generated features, this work provides the first topological data analysis of EMG-based gesture recognition for the characterization of the information encoded within a deep network, using handcrafted features as landmarks. This analysis reveals that handcrafted features and the learned features (in the earlier layers) both try to discriminate between all gestures, but do not encode the same information to do so. In the later layers, the learned features are inclined to instead adopt a one-vs.-all strategy for a given class. Furthermore, by using convolutional network visualization techniques, it is revealed that learned features actually tend to ignore the most activated channel during contraction, which is in stark contrast with the prevalence of handcrafted features designed to capture amplitude information. Overall, this work paves the way for hybrid feature sets by providing a clear guideline of complementary information encoded within learned and handcrafted features.

[1]  Vijayan K. Asari,et al.  The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches , 2018, ArXiv.

[2]  S. Sawilowsky New Effect Size Rules of Thumb , 2009 .

[3]  Angkoon Phinyomark,et al.  EMG feature evaluation for improving myoelectric pattern recognition robustness , 2013, Expert Syst. Appl..

[4]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[5]  S H Park,et al.  EMG pattern recognition based on artificial intelligence techniques. , 1998, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[6]  Bruce C. Wheeler,et al.  EMG feature evaluation for movement control of upper extremity prostheses , 1995 .

[7]  Benoit Gosselin,et al.  A Low-Cost, Wireless, 3-D-Printed Custom Armband for sEMG Hand Gesture Recognition , 2019, Sensors.

[8]  Erik Scheme,et al.  Linear Discriminant Analysis with Bayesian Risk Parameters for Myoelectric Control , 2019, 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[9]  Huosheng Hu,et al.  Support Vector Machine-Based Classification Scheme for Myoelectric Control Applied to Upper Limb , 2008, IEEE Transactions on Biomedical Engineering.

[10]  N. P. Reddy,et al.  Fractal analysis of surface EMG signals from the biceps. , 1997, International journal of medical informatics.

[11]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[12]  Mukund Balasubramanian,et al.  The Isomap Algorithm and Topological Stability , 2002, Science.

[13]  Pornchai Phukpattaranont,et al.  Probability Density Functions of Stationary Surface EMG Signals in Noisy Environments , 2016, IEEE Transactions on Instrumentation and Measurement.

[14]  Sijiang Du,et al.  Temporal vs. spectral approach to feature extraction from prehensile EMG signals , 2004, Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, 2004. IRI 2004..

[15]  Vidar Andersen,et al.  Electromyographic Comparison of Barbell Deadlift, Hex Bar Deadlift, and Hip Thrust Exercises: A Cross-Over Study , 2018, Journal of strength and conditioning research.

[16]  J. H. Ward Hierarchical Grouping to Optimize an Objective Function , 1963 .

[17]  L Guidetti,et al.  EMG patterns during running: Intra- and inter-individual variability. , 1996, Journal of electromyography and kinesiology : official journal of the International Society of Electrophysiological Kinesiology.

[18]  A E Grassino,et al.  Automatic assessment of electromyogram quality. , 1995, Journal of applied physiology.

[19]  François Laviolette,et al.  Domain-Adversarial Neural Networks , 2014, ArXiv.

[20]  Clément Gosselin,et al.  A convolutional neural network for robotic arm guidance using sEMG based frequency-features , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[21]  Trevor Darrell,et al.  Adversarial Discriminative Domain Adaptation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[23]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[24]  Andrea Vedaldi,et al.  Efficient Parametrization of Multi-domain Deep Neural Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Guangcan Liu,et al.  On fusing the latent deep CNN feature for image classification , 2018, World Wide Web.

[26]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[27]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[28]  Erik Scheme,et al.  A feature extraction issue for myoelectric control based on wearable EMG sensors , 2018, 2018 IEEE Sensors Applications Symposium (SAS).

[29]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Clément Gosselin,et al.  Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[31]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[32]  Erik Scheme,et al.  Navigating features: a topologically informed chart of electromyographic features space , 2017, Journal of The Royal Society Interface.

[33]  Huosheng Hu,et al.  Myoelectric control systems - A survey , 2007, Biomed. Signal Process. Control..

[34]  Lin Chen,et al.  NeuroEvolution of Augmenting Topologies with Learning for Data Classification , 2006, 2006 International Conference on Information and Automation.

[35]  Marek Malik,et al.  Individual patterns of QT/RR relationship. , 2002, Cardiac electrophysiology review.

[36]  A. Phinyomark,et al.  ELECTROMYOGRAPHY (EMG) SIGNAL CLASSIFICATION BASED ON DETRENDED FLUCTUATION ANALYSIS , 2011 .

[37]  Egon L. van den Broek,et al.  Computing Emotion Awareness Through Facial Electromyography , 2006, ECCV Workshop on HCI.

[38]  Blair A. Lock,et al.  Determining the Optimal Window Length for Pattern Recognition-Based Myoelectric Control: Balancing the Competing Effects of Classification Error and Controller Delay , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[39]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[40]  F. Wilcoxon Individual Comparisons by Ranking Methods , 1945 .

[41]  Yi Yang,et al.  DevNet: A Deep Event Network for multimedia event detection and evidence recounting , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Huosheng Hu,et al.  The effects of the force of contraction and elbow joint angle on mean and median frequency analysis for muscle fatigue evaluation , 2015 .

[43]  Facundo Mémoli,et al.  Topological Methods for the Analysis of High Dimensional Data Sets and 3D Object Recognition , 2007, PBG@Eurographics.

[44]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[45]  Heeyoul Choi,et al.  Robust kernel Isomap , 2007, Pattern Recognit..

[46]  Pornchai Phukpattaranont,et al.  Fractal analysis features for weak and single-channel upper-limb EMG signals , 2012, Expert Syst. Appl..

[47]  Pornchai Phukpattaranont,et al.  Feature reduction and selection for EMG signal classification , 2012, Expert Syst. Appl..

[48]  Manfredo Atzori,et al.  Deep Learning with Convolutional Neural Networks Applied to Electromyography Data: A Resource for the Classification of Movements for Prosthetic Hands , 2016, Front. Neurorobot..

[49]  Dinesh Kant Kumar,et al.  Decoding subtle forearm flexions using fractal features of surface electromyogram from single and multiple sensors , 2010, Journal of NeuroEngineering and Rehabilitation.

[50]  Abhishek Das,et al.  Grad-CAM: Why did you say that? , 2016, ArXiv.

[51]  Erik Scheme,et al.  EMG Pattern Recognition in the Era of Big Data and Deep Learning , 2018, Big Data Cogn. Comput..

[52]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[53]  Mark Halaki,et al.  Normalization of EMG Signals: To Normalize or Not to Normalize and What to Normalize to? , 2012 .

[54]  G. Carlsson,et al.  Topology based data analysis identifies a subgroup of breast cancers with a unique mutational profile and excellent survival , 2011, Proceedings of the National Academy of Sciences.

[55]  Yinfeng Fang,et al.  Surface electromyography feature extraction via convolutional neural network , 2020, Int. J. Mach. Learn. Cybern..

[56]  Chi-Woong Mun,et al.  Comparison of k-nearest neighbor, quadratic discriminant and linear discriminant analysis in classification of electromyogram signals based on the wrist-motion directions , 2011 .

[57]  Songfan Yang,et al.  Multi-scale Recognition with DAG-CNNs , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[58]  Giulio Sandini,et al.  Multi-subject/daily-life activity EMG-based control of mechanical hands , 2009, Journal of NeuroEngineering and Rehabilitation.

[59]  Adrian D. C. Chan,et al.  Identification of Contaminant Type in Surface Electromyography (EMG) Signals , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[60]  Erik Scheme,et al.  Electromyogram pattern recognition for control of powered upper-limb prostheses: state of the art and challenges for clinical use. , 2011, Journal of rehabilitation research and development.

[61]  Clément Gosselin,et al.  Transfer learning for sEMG hand gestures recognition using convolutional neural networks , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[62]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Stefano Ermon,et al.  A DIRT-T Approach to Unsupervised Domain Adaptation , 2018, ICLR.

[64]  Benoit Gosselin,et al.  Engaging with Robotic Swarms , 2019, ACM Transactions on Human-Robot Interaction.

[65]  Jed A. Meltzer,et al.  Individual differences in EEG theta and alpha dynamics during working memory correlate with fMRI responses across subjects , 2007, Clinical Neurophysiology.

[66]  Loris Nanni,et al.  Handcrafted vs. non-handcrafted features for computer vision classification , 2017, Pattern Recognit..

[67]  Kianoush Nazarpour,et al.  Combined influence of forearm orientation and muscular contraction on EMG pattern recognition , 2016, Expert Syst. Appl..

[68]  Yann LeCun,et al.  Pedestrian Detection with Unsupervised Multi-stage Feature Learning , 2012, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[69]  A. Chan,et al.  Fractal analysis of surface electromyography signals: a novel power spectrum-based method. , 2009, Journal of electromyography and kinesiology : official journal of the International Society of Electrophysiological Kinesiology.

[70]  J. Gitter,et al.  Fractal analysis of the electromyographic interference pattern , 1995, Journal of Neuroscience Methods.

[71]  Dario Farina,et al.  Multiday EMG-Based Classification of Hand Motions with Deep Learning Techniques , 2018, Sensors.

[72]  Desney S. Tan,et al.  Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces , 2008, CHI.

[73]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[74]  Olaf Sporns,et al.  Towards a new approach to reveal dynamical organization of the brain using topological data analysis , 2018, Nature Communications.

[75]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[76]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[77]  Pornchai Phukpattaranont,et al.  A Review of Control Methods for Electric Power Wheelchairs Based on Electromyography Signals with Special Emphasis on Pattern Recognition , 2011 .

[78]  Huosheng Hu,et al.  GA-based Feature Subset Selection for Myoelectric Classification , 2006, 2006 IEEE International Conference on Robotics and Biomimetics.

[79]  Sirinee Thongpanja,et al.  Mean and Median Frequency of EMG Signal to Determine Muscle Force based on Time- dependent Power Spectrum , 2013 .

[80]  Rami N. Khushaba,et al.  Differences in EMG Feature Space between Able-Bodied and Amputee Subjects for Myoelectric Control , 2019, 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

[81]  Zhang Qingju,et al.  Wavelet De-Noising of Electromyography , 2006, 2006 International Conference on Mechatronics and Automation.

[82]  Pornchai Phukpattaranont,et al.  Critical Exponent Analysis Applied to Surface EMG Signals for Gesture Recognition , 2011 .

[83]  Bohyung Han,et al.  Learning Multi-domain Convolutional Neural Networks for Visual Tracking , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[84]  Olaf Sporns,et al.  Generating dynamical neuroimaging spatiotemporal representations (DyNeuSR) using topological data analysis , 2019, Network Neuroscience.

[85]  Guido Bugmann,et al.  Improving the Performance Against Force Variation of EMG Controlled Multifunctional Upper-Limb Prostheses for Transradial Amputees , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[86]  Tianqi Chen,et al.  Empirical Evaluation of Rectified Activations in Convolutional Network , 2015, ArXiv.

[87]  Jiaying Liu,et al.  Revisiting Batch Normalization For Practical Domain Adaptation , 2016, ICLR.

[88]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[89]  Erik Scheme,et al.  Feature Extraction and Selection for Pain Recognition Using Peripheral Physiological Signals , 2019, Front. Neurosci..

[90]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[91]  Kongqiao Wang,et al.  Hand gesture recognition and virtual game control based on 3D accelerometer and EMG sensors , 2009, IUI.

[92]  Erik Cambria,et al.  Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis , 2015, EMNLP.

[93]  F Grizzi,et al.  Fractal analysis. , 2000, Gynecologic oncology.

[94]  Yongxin Yang,et al.  A Unified Perspective on Multi-Domain and Multi-Task Learning , 2014, ICLR.