A Multimodal Framework Based on Integration of Cortical and Muscular Activities for Decoding Human Intentions About Lower Limb Motions

In this study, a multimodal fusion framework based on three different modal biosignals is developed to recognize human intentions related to lower limb multi-joint motions which commonly appear in daily life. Electroencephalogram (EEG), electromyogram (EMG) and mechanomyogram (MMG) signals were simultaneously recorded from twelve subjects while performing nine lower limb multi-joint motions. These multimodal data are used as the inputs of the fusion framework for identification of different motion intentions. Twelve fusion techniques are evaluated in this framework and a large number of comparative experiments are carried out. The results show that a support vector machine-based three-modal fusion scheme can achieve average accuracies of 98.61%, 97.78% and 96.85%, respectively, under three different data division forms. Furthermore, the relevant statistical tests reveal that this fusion scheme brings significant accuracy improvement in comparison with the cases of two-modal fusion or only a single modality. These promising results indicate the potential of the multimodal fusion framework for facilitating the future development of human-robot interaction for lower limb rehabilitation.

[1]  Hakan Toreyin,et al.  A Low-Power ASIC Signal Processor for a Vestibular Prosthesis , 2016, IEEE Transactions on Biomedical Circuits and Systems.

[2]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[3]  Takao Someya,et al.  1 $\mu$m-Thickness Ultra-Flexible and High Electrode-Density Surface Electromyogram Measurement Sheet With 2 V Organic Transistors for Prosthetic Hand Control , 2014, IEEE Transactions on Biomedical Circuits and Systems.

[4]  Evgeny Bogdanov,et al.  Multifunctional Neurodevice for Recognition of Electrophysiological Signals and Data Transmission in an Exoskeleton Construction , 2016 .

[5]  Gamini Dissanayake,et al.  Driver Drowsiness Classification Using Fuzzy Wavelet-Packet-Based Feature-Extraction Algorithm , 2011, IEEE Transactions on Biomedical Engineering.

[6]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[7]  L. J. Chapman,et al.  The measurement of foot preference , 1987, Neuropsychologia.

[8]  Mohan S. Kankanhalli,et al.  Multimodal fusion for multimedia analysis: a survey , 2010, Multimedia Systems.

[9]  Hoi-Jun Yoo,et al.  A Wearable Neuro-Feedback System With EEG-Based Mental Status Monitoring and Transcranial Electrical Stimulation , 2014, IEEE Transactions on Biomedical Circuits and Systems.

[10]  C. Neuper,et al.  Combining Brain–Computer Interfaces and Assistive Technologies: State-of-the-Art and Challenges , 2010, Front. Neurosci..

[11]  Wonkeun Youn,et al.  Feasibility of using an artificial neural network model to estimate the elbow flexion force from mechanomyography , 2011, Journal of Neuroscience Methods.

[12]  Marek Kurzynski,et al.  Meta-Bayes Classifier with Markov Model Applied to the Control of Bioprosthetic Hand , 2016 .

[13]  Galina L. Rogova,et al.  Reliability In Information Fusion : Literature Survey , 2004 .

[14]  Enrique Mario Spinelli,et al.  Analysis and Simple Circuit Design of Double Differential EMG Active Electrode. , 2016, IEEE transactions on biomedical circuits and systems.

[15]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Chun-Hsiang Chuang,et al.  Wireless and Wearable EEG System for Evaluating Driver Vigilance , 2014, IEEE Transactions on Biomedical Circuits and Systems.

[17]  Luca Benini,et al.  A Versatile Embedded Platform for EMG Acquisition and Gesture Recognition , 2015, IEEE Transactions on Biomedical Circuits and Systems.

[18]  Ricardo Chavarriaga,et al.  A hybrid brain–computer interface based on the fusion of electroencephalographic and electromyographic activities , 2011, Journal of neural engineering.

[19]  Yoshiaki Hayashi,et al.  A study of EMG and EEG during perception-assist with an upper-limb power-assist robot , 2012, 2012 IEEE International Conference on Robotics and Automation.

[20]  Xinjun Sheng,et al.  Development of a Hybrid Surface EMG and MMG Acquisition System for Human Hand Motion Analysis , 2015, ICIRA.

[21]  Robert Riener,et al.  A survey of sensor fusion methods in wearable robotics , 2015, Robotics Auton. Syst..

[22]  Michael Goldfarb,et al.  An Approach for the Cooperative Control of FES With a Powered Exoskeleton During Level Walking for Persons With Paraplegia , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[23]  Yan Song,et al.  A synchronous and multi-domain feature extraction method of EEG and sEMG in power-assist rehabilitation robot , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[24]  Roozbeh Jafari,et al.  Design Principles and Dynamic Front End Reconfiguration for Low Noise EEG Acquisition With Finger Based Dry Electrodes , 2015, IEEE Transactions on Biomedical Circuits and Systems.

[25]  Roberto Guerrieri,et al.  Active Electrode IC for EEG and Electrical Impedance Tomography With Continuous Monitoring of Contact Impedance , 2015, IEEE Transactions on Biomedical Circuits and Systems.

[26]  Xiaoling Chen,et al.  Identification Method of Human Movement Intention based on the Fusion Feature of EEG and EMG , .

[27]  R. Buschbacher.  Anatomical Guide for the Electromyographer: The Limbs and Trunk , 2007 .

[28]  Pornchai Phukpattaranont,et al.  Feature reduction and selection for EMG signal classification , 2012, Expert Syst. Appl..

[29]  S. Coren.  The lateral preference inventory for measurement of handedness, footedness, eyedness, and earedness: Norms for young adults , 1993 .

[30]  Kevin B. Englehart,et al.  A wavelet-based continuous classification scheme for multifunction myoelectric control , 2001, IEEE Transactions on Biomedical Engineering.

[31]  D. Altman,et al.  Multiple significance tests: the Bonferroni method , 1995, BMJ.

[32]  C. Orizio.  Muscle sound: bases for the introduction of a mechanomyographic signal in muscle studies. , 1993, Critical reviews in biomedical engineering.

[33]  R.F. Weir,et al.  The Optimal Controller Delay for Myoelectric Prostheses , 2007, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[34]  Joaquim Filipe,et al.  Identification of Hand Movements based on MMG and EMG Signals , 2008, BIOSIGNALS.

[35]  Kevin B. Englehart,et al.  A robust, real-time control scheme for multifunction myoelectric control , 2003, IEEE Transactions on Biomedical Engineering.

[36]  Charles Sodini,et al.  Low-Power, 8-Channel EEG Recorder and Seizure Detector ASIC for a Subdermal Implantable System , 2016, IEEE Transactions on Biomedical Circuits and Systems.

[37]  Shiqian Wang,et al.  Design and Control of the MINDWALKER Exoskeleton , 2015, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[38]  Jan Van der Spiegel,et al.  The PennBMBI: Design of a General Purpose Wireless Brain-Machine-Brain Interface System , 2015, IEEE Transactions on Biomedical Circuits and Systems.

[39]  Daniel P. Ferris,et al.  Locomotor Adaptation by Transtibial Amputees Walking With an Experimental Powered Prosthesis Under Continuous Myoelectric Control , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[40]  Marek Kurzynski,et al.  Control System of Bioprosthetic Hand Based on Advanced Analysis of Biosignals and Feedback from the Prosthesis Sensors , 2012, ITIB.

[41]  Nuria Rodríguez,et al.  Flexible Communication and Control Protocol for Injectable Neuromuscular Interfaces , 2007, IEEE Transactions on Biomedical Circuits and Systems.

[42]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[43]  Qing He,et al.  Motion intent recognition of individual fingers based on mechanomyogram , 2017, Pattern Recognit. Lett..

[44]  Marek Kurzynski,et al.  Multiclassifier System with Dynamic Model of Classifier Competence Applied to the Control of Bioprosthetic Hand , 2015, GCAI.