Virtual Reality to Study the Gap Between Offline and Real-Time EMG-based Gesture Recognition

Within sEMG-based gesture recognition, a chasm exists in the literature between offline accuracy and real-time usability of a classifier. This gap mainly stems from the four main dynamic factors in sEMG-based gesture recognition: gesture intensity, limb position, electrode shift and transient changes in the signal. These factors are hard to include within an offline dataset as each of them exponentially augment the number of segments to be recorded. On the other hand, online datasets are biased towards the sEMG-based algorithms providing feedback to the participants, limiting the usability of such datasets as benchmarks. This paper proposes a virtual reality (VR) environment and a real-time experimental protocol from which the four main dynamic factors can more easily be studied. During the online experiment, the gesture recognition feedback is provided through the leap motion camera, enabling the proposed dataset to be re-used to compare future sEMG-based algorithms. 20 able-bodied persons took part in this study, completing three to four sessions over a period spanning between 14 and 21 days. Finally, TADANN, a new transfer learning-based algorithm, is proposed for long term gesture classification and significantly (p<0.05) outperforms fine-tuning a network.

[1]  Benoit Gosselin,et al.  Engaging with Robotic Swarms , 2019, ACM Transactions on Human-Robot Interaction.

[2]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[3]  Erik Scheme,et al.  Electromyogram pattern recognition for control of powered upper-limb prostheses: state of the art and challenges for clinical use. , 2011, Journal of rehabilitation research and development.

[4]  Dario Farina,et al.  Multiday EMG-Based Classification of Hand Motions with Deep Learning Techniques , 2018, Sensors.

[5]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[6]  Clément Gosselin,et al.  Transfer learning for sEMG hand gestures recognition using convolutional neural networks , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[7]  Adel Al-Jumaily,et al.  Electromyogram (EMG) driven system based virtual reality for prosthetic and rehabilitation devices , 2009, iiWAS.

[8]  Theocharis Kyriacou,et al.  Feasibility of using combined EMG and kinematic signals for prosthesis control: A simulation study using a virtual reality environment , 2016, Journal of electromyography and kinesiology : official journal of the International Society of Electrophysiological Kinesiology.

[9]  Nitish V. Thakor,et al.  User Training for Pattern Recognition-Based Myoelectric Prostheses: Improving Phantom Limb Movement Consistency and Distinguishability , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[10]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[11]  Andrew Jackson,et al.  Learning a Novel Myoelectric-Controlled Interface Task , 2008, Journal of neurophysiology.

[12]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[13]  Kianoush Nazarpour,et al.  Artificial Proprioceptive Feedback for Myoelectric Control , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[14]  Zhang Xia,et al.  EMG-driven computer game for post-stroke rehabilitation , 2010, 2010 IEEE Conference on Robotics, Automation and Mechatronics.

[15]  Tianqi Chen,et al.  Empirical Evaluation of Rectified Activations in Convolutional Network , 2015, ArXiv.

[16]  Erik J. Scheme,et al.  Confidence-Based Rejection for Improved Pattern Recognition Myoelectric Control , 2013, IEEE Transactions on Biomedical Engineering.

[17]  Yu Liu,et al.  CNN-RNN: a large-scale hierarchical image classification framework , 2018, Multimedia Tools and Applications.

[18]  José Luis Pons Rovira,et al.  Virtual reality training and EMG control of the MANUS hand prosthesis , 2005, Robotica.

[19]  Benoit Gosselin,et al.  Interpreting Deep Learning Features for Myoelectric Control: A Comparison With Handcrafted Features , 2020, Frontiers in Bioengineering and Biotechnology.

[20]  Giulio Sandini,et al.  Multi-subject/daily-life activity EMG-based control of mechanical hands , 2009, Journal of NeuroEngineering and Rehabilitation.

[21]  François Laviolette,et al.  Domain-Adversarial Neural Networks , 2014, ArXiv.

[22]  Erik Scheme,et al.  A feature extraction issue for myoelectric control based on wearable EMG sensors , 2018, 2018 IEEE Sensors Applications Symposium (SAS).

[23]  Clément Gosselin,et al.  Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[24]  Huosheng Hu,et al.  Myoelectric control systems - A survey , 2007, Biomed. Signal Process. Control..

[25]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[26]  Yongkang Wong,et al.  A novel attention-based hybrid CNN-RNN architecture for sEMG-based gesture recognition , 2018, PloS one.

[27]  Vladlen Koltun,et al.  An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling , 2018, ArXiv.

[28]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[29]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[30]  Clément Gosselin,et al.  A convolutional neural network for robotic arm guidance using sEMG based frequency-features , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[31]  B Hudgins,et al.  Myoelectric signal processing for control of powered limb prostheses. , 2006, Journal of electromyography and kinesiology : official journal of the International Society of Electrophysiological Kinesiology.

[32]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[33]  Erik Scheme,et al.  Real-time, simultaneous myoelectric control using a convolutional neural network , 2018, PloS one.

[34]  Blair A. Lock,et al.  Determining the Optimal Window Length for Pattern Recognition-Based Myoelectric Control: Balancing the Competing Effects of Classification Error and Controller Delay , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[35]  S. Sawilowsky.  New Effect Size Rules of Thumb , 2009 .

[36]  Angkoon Phinyomark,et al.  EMG feature evaluation for improving myoelectric pattern recognition robustness , 2013, Expert Syst. Appl..

[37]  A.D.C. Chan,et al.  Examining the adverse effects of limb position on pattern recognition based myoelectric control , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[38]  Yuanliu Liu,et al.  Video-based emotion recognition using CNN-RNN and C3D hybrid networks , 2016, ICMI.

[39]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[40]  Arto Visala,et al.  urrent state of digital signal processing in myoelectric interfaces and elated applications , 2015 .

[41]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[42]  Frank Weichert,et al.  Analysis of the Accuracy and Robustness of the Leap Motion Controller , 2013, Sensors.

[43]  Heiga Zen,et al.  WaveNet: A Generative Model for Raw Audio , 2016, SSW.

[44]  Benoit Gosselin,et al.  A Low-Cost, Wireless, 3-D-Printed Custom Armband for sEMG Hand Gesture Recognition , 2019, Sensors.