Human emotion detection through speech and facial expressions

Estimation of human emotions using a computer has been difficult since when human engaged in a conversational secession. In this research work, a proposed hybrid system through facial expressions and speech is used to estimate basic emotions (angry, sad, happy, boredom, disgust and surprise) of a person when he is engaged in a conversational secession. Relative Bin Frequency Coefficients and Relative Sub-Image Based features are used for acoustic and visual data respectively. Support Vector Machine with radial basis kernel is used for classification. This research work revealed that the proposed feature extraction through speech and facial expression is the most prominent aspect affecting the emotion detection system accompanied by proposed fusion technique. Although, some aspects considered affecting the emotion detection system, this affect is relatively minor. It was observed that the performance of the bimodal emotion detection system is low than the unimodal emotion detection system through deliberate facial expressions. In order to deal with the issue, a suitable database is used. The results indicated that the proposed emotion detection system showed better performance with respect to basic emotional classes than the rest.

[1]  M. Bartlett,et al.  Machine Analysis of Facial Expressions , 2007 .

[2]  Christine L. Lisetti,et al.  Toward multimodal fusion of affective cues , 2006, HCM '06.

[3]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[4]  Abas Md Said,et al.  Emotion detection using sub-image based features through human facial expressions , 2012, 2012 International Conference on Computer & Information Science (ICCIS).

[5]  M. Y. Nayan,et al.  Emotion detection using relative amplitude-based features through speech , 2012, 2012 International Conference on Computer & Information Science (ICCIS).

[6]  Marc Hanheide,et al.  Evaluation and Discussion of Multi-modal Emotion Recognition , 2009, 2009 Second International Conference on Computer and Electrical Engineering.

[7]  Fabio Pianesi,et al.  A first evaluation study of a database of kinetic facial expressions (DaFEx) , 2005, ICMI '05.

[8]  Jun Wang,et al.  3D Facial Expression Recognition Based on Primitive Surface Feature Distribution , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[9]  Peter Robinson,et al.  Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[10]  T. Shaw.  Darwin on the Expression of the Emotions in Man and Animals , 1873 .

[11]  Maja Pantic,et al.  Fully automatic facial feature point detection using Gabor feature based boosted classifiers , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[12]  Klaus R. Scherer,et al.  Vocal communication of emotion , 2000 .

[13]  J. Lien,et al.  Automatic recognition of facial expressions using hidden markov models and estimation of expression intensity , 1998 .

[14]  Chun Chen,et al.  Speech Emotion Recognition Based on a Fusion of All-Class and Pairwise-Class Feature Selection , 2007, International Conference on Computational Science.

[15]  R. Plutchik.  A GENERAL PSYCHOEVOLUTIONARY THEORY OF EMOTION , 1980 .

[16]  Xiaohu Wang,et al.  Multimodal Emotion Recognition Integrating Affective Speech with Facial Expression , 2014 .

[17]  Stan Z. Li,et al.  Statistical Learning of Evaluation Function for ASM/AAM Image Alignment , 2004, ECCV Workshop BioAW.

[18]  Rahul Sathawane,et al.  Analysis of Emotion Recognition using Facial Expressions, using Bezier curve , 2015 .

[19]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[20]  J. Russell,et al.  Facial and vocal expressions of emotion. , 2003, Annual review of psychology.

[21]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[22]  D. Massaro,et al.  Perceiving affect from the voice and the face , 1996, Psychonomic bulletin & review.

[23]  Yi-Cheng Zhang,et al.  Using SVM to design facial expression recognition for shape and texture features , 2010, 2010 International Conference on Machine Learning and Cybernetics.

[24]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[25]  Garrison W. Cottrell,et al.  Representing Face Images for Emotion Classification , 1996, NIPS.

[26]  Jun Ohya,et al.  Spotting segments displaying facial expression from image sequences using HMM , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[27]  Eugen Lupu,et al.  Emotions recognition by speechand facial expressions analysis , 2009, 2009 17th European Signal Processing Conference.