Expression Recognition by Using Facial and Vocal Expressions

Human behaviour may be monitored by analysing facial expressions and vocal expressions. Hence an automatic technique which combines both these features will give a more accurate overall estimation of expression. In this work we propose a new method which is uses facial and vocal features to estimate the expression of the subject. Facial expressions are analysed by extracting important facial features and then clustering the movement of these features. In parallel the voice is processed by using considering sudden changes in amplitude and frequency in order to recognize the expression. Finally a weighted sum rule is used to combine the decisions obtained by facial and vocal expression recognition. The proposed technique is tested on an ongoing set of real data monitored by a psychologist.

[1]  Mohammed Yeasin,et al.  From facial expression to level of interest: a spatio-temporal approach , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[2]  Jing Xiao,et al.  Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[3]  Hongying Meng,et al.  Affective State Level Recognition in Naturalistic Facial and Vocal Expressions , 2014, IEEE Transactions on Cybernetics.

[4]  Albino Nogueiras,et al.  Speech emotion recognition using hidden Markov models , 2001, INTERSPEECH.

[5]  Samir Benbelkacem,et al.  Integrating human-computer interaction and business practices for mixed reality systems design: a case study , 2014, IET Softw..

[6]  Ling Guan,et al.  A Deformable 3-D Facial Expression Model for Dynamic Human Emotional State Recognition , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[7]  Z. Obrenovic,et al.  Modeling multimodal human-computer interaction , 2004, Computer.

[8]  Stefanos Zafeiriou,et al.  Audiovisual classification of vocal outbursts in human conversation using Long-Short-Term Memory networks , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[9]  Zakia Hammal,et al.  Pain monitoring: A dynamic and context-sensitive system , 2012, Pattern Recognit..

[10]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[11]  Gwen Littlewort,et al.  Automatic Recognition of Facial Actions in Spontaneous Expressions , 2006, J. Multim..

[12]  Cagatay Basdogan,et al.  Intention Recognition for Dynamic Role Exchange in Haptic Collaboration , 2013, IEEE Transactions on Haptics.

[13]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.