Eyebrow emotional expression recognition using surface EMG signals

The main objective of this study is to recognize facial emotional expression effectively in human-computer interaction. A surface electromyography (sEMG) based eyebrow emotional expression recognition method is proposed. Using a specially designed headband, we conducted an experiment in which we recorded the sEMG signals from the frontalis and corrugator supercilii muscles of six participants who were instructed to pose the facial expressions of anger, fear, sadness, surprise and disgust. Subsequently, six features of the sEMG time domain were extracted and used as input vectors to an emotion recognition model based on an Elman neural network (ENN). The performance of this model was compared to another recognition model based on a Back Propagation neural network (BPNN). The average recognition rate for the five emotions achieved by the ENN-based model was 97.12% in the training and 96.12% in the test set, which was slightly superior to the performance of the BPNN-based model.

[1]  Gilles Pourtois,et al.  Simultaneous recording of EEG and facial muscle reactions during spontaneous emotional mimicry , 2008, Neuropsychologia.

[2]  Larry S. Davis,et al.  Human expression recognition from motion using a radial basis function network architecture , 1996, IEEE Trans. Neural Networks.

[3]  Horst-Michael Groß,et al.  A real-time facial expression recognition system based on Active Appearance Models using gray images and edge images , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[4]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  R. Gur,et al.  Differences in facial expressions of four universal emotions , 2004, Psychiatry Research.

[6]  Nadia Bianchi-Berthouze,et al.  Emotion recognition by two view SVM_2K classifier on dynamic facial expression features , 2011, Face and Gesture 2011.

[7]  D. Stegeman,et al.  A surface EMG electrode for the simultaneous observation of multiple facial muscles , 2003, Journal of Neuroscience Methods.

[8]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[9]  Garrison W. Cottrell,et al.  Representing Face Images for Emotion Classification , 1996, NIPS.

[10]  Qiang Ji,et al.  Facial expression understanding in image sequences using dynamic and active visual information fusion , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[11]  Y. V. Venkatesh,et al.  Facial expression recognition using radial encoding of local Gabor features and classifier synthesis , 2012, Pattern Recognit..

[12]  Sandra Miener.  Die Basisemotion Ekel : Untersuchungen zum Zusammenhang zwischen Gefühl und Ausdruck , 2007 .

[13]  Takeo Kanade,et al.  Facial Expression Recognition , 2011, Handbook of Face Recognition.

[14]  Karen L. Schmidt,et al.  Comparison of Deliberate and Spontaneous Facial Movement in Smiles and Eyebrow Raises , 2009, Journal of nonverbal behavior.

[15]  Zhu Wang,et al.  A research using hybrid RBF/Elman neural networks for intrusion detection system secure model , 2009, Comput. Phys. Commun..

[16]  Shi Chen,et al.  Emotion-driven Chinese folk music-image retrieval based on DE-SVM , 2015, Neurocomputing.

[17]  Jeffrey F. Cohn,et al.  Dynamics of facial expression: normative characteristics and individual differences , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[18]  Bing Zhou,et al.  Facial expression cloning with elastic and muscle models , 2014, J. Vis. Commun. Image Represent..

[19]  Robert D. Ward,et al.  Physiological responses to different WEB page designs , 2003, Int. J. Hum. Comput. Stud..

[20]  Gwen Littlewort,et al.  An approach to automatic recognition of spontaneous facial actions , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[21]  Maja Pantic,et al.  Expert system for automatic analysis of facial expressions , 2000, Image Vis. Comput..

[22]  Nicu Sebe,et al.  Facial expression recognition from video sequences , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[23]  Kejun Zhang,et al.  An Upper-Limb Power-Assist Exoskeleton Using Proportional Myoelectric Control , 2014, Sensors.

[24]  G. Horstmann,et al.  Coherence between Emotion and Facial Expression: Evidence from Laboratory Experiments , 2013 .

[25]  Jin Hui,et al.  Analysis and Application of the Facial Expression Motions Based on Eigen-Flow , 2003 .

[26]  Zhengyou Zhang,et al.  Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[27]  A. J. Fridlund,et al.  Pattern recognition of self-reported emotional state from multiple-site facial EMG activity during affective imagery. , 1984, Psychophysiology.

[28]  Qijun Zhao,et al.  Facial expression recognition on multiple manifolds , 2011, Pattern Recognit..

[29]  Marcelo M. Wanderley,et al.  Common cues to emotion in the dynamic facial expressions of speech and song , 2014, Quarterly journal of experimental psychology.

[30]  Shaogang Gong,et al.  Robust facial expression recognition using local binary patterns , 2005, IEEE International Conference on Image Processing 2005.

[31]  Changmok Choi,et al.  Noninvasive sEMG-based control for humanoid robot teleoperated navigation , 2011 .

[32]  Othman Omran Khalifa,et al.  EMG signal classification for human computer interaction: a review , 2009 .

[33]  Mélanie Perron,et al.  Featural processing in recognition of emotional facial expressions , 2014, Cognition & emotion.

[34]  Richard L. Hazlett,et al.  Measuring emotional valence to understand the user's experience of software , 2007, Int. J. Hum. Comput. Stud..

[35]  Gwen Littlewort,et al.  Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction. , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[36]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Takeo Kanade,et al.  Feature-point tracking by optical flow discriminates subtle differences in facial expression , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[38]  Murphy-ChutorianErik,et al.  Head Pose Estimation in Computer Vision , 2009 .

[39]  Jonathan Klein,et al.  Frustrating the user on purpose: a step toward building an affective computer , 2002, Interact. Comput..

[40]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[41]  M. Thüring,et al.  Usability, aesthetics and emotions in human–technology interaction , 2007 .

[42]  Sougata Karmakar,et al.  Muscle Computer Interface: A Review , 2013 .

[43]  Saeed Zolfaghari,et al.  Chaotic time series prediction with residual analysis method using hybrid Elman-NARX neural networks , 2010, Neurocomputing.

[44]  N. Georganas,et al.  Human emotion recognition from body language of the head using soft computing techniques , 2012 .

[45]  J. Russell.  Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. , 1994, Psychological bulletin.

[46]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[47]  A. J. Fridlund,et al.  Guidelines for human electromyographic research. , 1986, Psychophysiology.

[48]  M. Pietikäinen,et al.  Facial expression recognition based on local binary patterns , 2007, Pattern Recognition and Image Analysis.

[49]  Michael Isard,et al.  ICONDENSATION: Unifying Low-Level and High-Level Tracking in a Stochastic Framework , 1998, ECCV.

[50]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression (PIE) database , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[51]  P. Ekman,et al.  EMFACS-7: Emotional Facial Action Coding System , 1983 .

[52]  Rogério Schmidt Feris,et al.  Manifold Based Analysis of Facial Expression , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[53]  Jalal Shiri,et al.  Artificial neural networks vs. Gene Expression Programming for estimating outlet dissolved oxygen in micro-irrigation sand filters fed with effluents , 2013 .

[54]  P. Ekman.  Emotion in the human face , 1982 .

[55]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..