Deep Learning with Convolutional Neural Networks Applied to Electromyography Data: A Resource for the Classification of Movements for Prosthetic Hands

Natural control methods based on surface electromyography (sEMG) and pattern recognition are promising for hand prosthetics. However, the control robustness offered by scientific research is still not sufficient for many real life applications, and commercial prostheses are capable of offering natural control for only a few movements. In recent years deep learning revolutionized several fields of machine learning, including computer vision and speech recognition. Our objective is to test its methods for natural control of robotic hands via sEMG using a large number of intact subjects and amputees. We tested convolutional networks for the classification of an average of 50 hand movements in 67 intact subjects and 11 transradial amputees. The simple architecture of the neural network allowed to make several tests in order to evaluate the effect of pre-processing, layer architecture, data augmentation and optimization. The classification results are compared with a set of classical classification methods applied on the same datasets. The classification accuracy obtained with convolutional neural networks using the proposed architecture is higher than the average results obtained with the classical classification methods, but lower than the results obtained with the best reference methods in our tests. The results show that convolutional neural networks with a very simple architecture can produce accurate results comparable to the average classical classification methods. They show that several factors (including pre-processing, the architecture of the net and the optimization parameters) can be fundamental for the analysis of sEMG data. Larger networks can achieve higher accuracy on computer vision and object recognition tasks. This fact suggests that it may be interesting to evaluate if larger networks can increase sEMG classification accuracy too.

[1]  Bruce C. Wheeler,et al.  EMG feature evaluation for movement control of upper extremity prostheses , 1995 .

[2]  S. Edwards,et al.  Developmental and functional hand grasps , 2002 .

[3]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[4]  Nitish V. Thakor,et al.  Decoding of Individuated Finger Movements Using Surface Electromyography , 2009, IEEE Transactions on Biomedical Engineering.

[5]  Thomas Feix,et al.  A comprehensive grasp taxonomy , 2009 .

[6]  Navdeep Jaitly,et al.  Multi-task Neural Networks for QSAR Predictions , 2014, ArXiv.

[7]  Jana D. Moon,et al.  Quantification of Regenerative Peripheral Nerve Interface Signal Transmission , 2012 .

[8]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[9]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[10]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[11]  Rajesh P. N. Rao,et al.  Real-Time Classification of Electromyographic Signals for Robotic Control , 2005, AAAI.

[12]  Kevin B. Englehart,et al.  A robust, real-time control scheme for multifunction myoelectric control , 2003, IEEE Transactions on Biomedical Engineering.

[13]  Yann LeCun,et al.  Pedestrian Detection with Unsupervised Multi-stage Feature Learning , 2012, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  David G. Stork,et al.  Pattern Classification , 1973 .

[15]  Ilja Kuzborskij,et al.  Characterization of a Benchmark Database for Myoelectric Movement Classification , 2015, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[16]  Ilja Kuzborskij,et al.  On the challenge of classifying 52 hand movements from surface electromyography , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[17]  B. Hudgins,et al.  A Real-Time Pattern Recognition Based Myoelectric Control Usability Study Implemented in a Virtual Environment , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[18]  Guanglin Li,et al.  Motion recognition for simultaneous control of multifunctional transradial prostheses , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[19]  Bo Håkansson,et al.  Evaluation of classifier topologies for the real-time classification of simultaneous limb motions , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[20]  Pierre Baldi,et al.  Deep autoencoder neural networks for gene ontology annotation predictions , 2014, BCB.

[21]  Carlo J. De Luca,et al.  The Use of Surface Electromyography in Biomechanics , 1997 .

[22]  Manfredo Atzori,et al.  Electromyography data for non-invasive naturally-controlled robotic hand prostheses , 2014, Scientific Data.

[23]  Barbara Caputo,et al.  Natural control capabilities of robotic hands by hand amputated subjects , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[24]  Seong-Whan Lee,et al.  Movement intention decoding based on deep learning for multiuser myoelectric interfaces , 2016, 2016 4th International Winter Conference on Brain-Computer Interface (BCI).

[25]  Erik Scheme,et al.  Electromyogram pattern recognition for control of powered upper-limb prostheses: state of the art and challenges for clinical use. , 2011, Journal of rehabilitation research and development.

[26]  Sethu Vijayakumar,et al.  Evaluation of regression methods for the continuous decoding of finger movement from surface EMG and accelerometry , 2015, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER).

[27]  Toshio Tsuji,et al.  A human-assisting manipulator teleoperated by EMG signals and arm motions , 2003, IEEE Trans. Robotics Autom..

[28]  Jürgen Schmidhuber,et al.  Multi-column deep neural network for traffic sign classification , 2012, Neural Networks.

[29]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[30]  Vijay S. Pande,et al.  Massively Multitask Networks for Drug Discovery , 2015, ArXiv.

[31]  Mark R. Cutkosky,et al.  On grasp choice, grasp models, and the design of hands for manufacturing tasks , 1989, IEEE Trans. Robotics Autom..

[32]  Alicia J. Davis,et al.  Surveying the interest of individuals with upper limb loss in novel prosthetic control techniques , 2015, Journal of NeuroEngineering and Rehabilitation.

[33]  Geoffrey E. Hinton,et al.  Binary coding of speech spectrograms using a deep auto-encoder , 2010, INTERSPEECH.

[34]  Fei-Fei Li,et al.  What Does Classifying More Than 10, 000 Image Categories Tell Us? , 2010, ECCV.

[35]  Manfredo Atzori,et al.  Control Capabilities of Myoelectric Robotic Prostheses by Hand Amputees: A Scientific Research and Market Overview , 2015, Front. Syst. Neurosci..

[36]  Claudio Castellini,et al.  FFLS: An accurate linear device for measuring synergistic finger contractions , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[37]  Hiroshi Yokoi,et al.  Competitive Learning Method for Robust EMG-to-Motion Classifier , 2006, IAS.

[38]  O. Stavdahl,et al.  Control of Upper Limb Prostheses: Terminology and Proportional Myoelectric Control—A Review , 2012, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[39]  Dario Farina,et al.  Stereovision and augmented reality for closed-loop control of grasping in hand prostheses , 2014, Journal of neural engineering.

[40]  Levi J. Hargrove,et al.  Classification of Simultaneous Movements Using Surface EMG Pattern Recognition , 2013, IEEE Transactions on Biomedical Engineering.

[41]  Stefano Stramigioli,et al.  Myoelectric forearm prostheses: state of the art from a user-centered perspective. , 2011, Journal of rehabilitation research and development.

[42]  Harris Drucker,et al.  Comparison of learning algorithms for handwritten digit recognition , 1995 .

[43]  Geoffrey E. Hinton,et al.  Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine , 2010, NIPS.

[44]  G. Lundborg,et al.  Refined myoelectric control in below-elbow amputees using artificial neural networks and a data glove. , 2005, The Journal of hand surgery.

[45]  K. Englehart,et al.  Classification of the myoelectric signal using time-frequency based representations. , 1999, Medical engineering & physics.

[46]  Shigeo Abe DrEng Pattern Classification , 2001, Springer London.

[47]  Vikash Gilja,et al.  Long-term Stability of Neural Prosthetic Control Signals from Silicon Cortical Arrays in Rhesus Macaque Motor Cortex , 2010 .

[48]  Robert D. Lipschutz,et al.  Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. , 2009, JAMA.

[49]  Dario Farina,et al.  The Extraction of Neural Information from the Surface EMG for the Control of Upper-Limb Prostheses: Emerging Avenues and Challenges , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[50]  Manfredo Atzori,et al.  Movement Error Rate for Evaluation of Machine Learning Methods for sEMG-Based Hand Movement Classification , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[51]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[52]  A. M. Hager,et al.  Effect of clinical parameters on the control of myoelectric robotic prosthetic hands. , 2016, Journal of rehabilitation research and development.

[53]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[54]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[55]  Strahinja Došen,et al.  Cognitive vision system for control of dexterous prosthetic hands: Experimental evaluation , 2010, Journal of NeuroEngineering and Rehabilitation.

[56]  Giulio Sandini,et al.  Multi-subject/daily-life activity EMG-based control of mechanical hands , 2009, Journal of NeuroEngineering and Rehabilitation.

[57]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[58]  Patrick van der Smagt,et al.  Surface EMG in advanced hand prosthetics , 2008, Biological Cybernetics.

[59]  Henning Müller,et al.  Overview of the CLEF 2009 Medical Image Retrieval Track , 2009, CLEF.

[60]  N. Kamakura,et al.  Patterns of static prehension in normal hands. , 1980, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.

[61]  T. Kuiken,et al.  Quantifying Pattern Recognition—Based Myoelectric Control of Multifunctional Transradial Prostheses , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[62]  Andrea Vedaldi,et al.  MatConvNet: Convolutional Neural Networks for MATLAB , 2014, ACM Multimedia.

[63]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[64]  Marie-Françoise Lucas,et al.  Multi-channel surface EMG classification using support vector machines and signal-based wavelet optimization , 2008, Biomed. Signal Process. Control..

[65]  Barbara Caputo,et al.  Classification of hand movements in amputated subjects by sEMG and accelerometers , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[66]  Brian Kingsbury,et al.  New types of deep neural network learning for speech recognition and related applications: an overview , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[67]  Dario Farina,et al.  Is Accurate Mapping of EMG Signals on Kinematics Needed for Precise Online Myoelectric Control? , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.