Combining Empirical Studies of Audio-Lingual and Visual-Facial Modalities for Emotion Recognition

In this paper, we present and discuss two empirical studies that we have conducted involving human subjects and human observers concerning the recognition of emotions from audio-lingual and visual-facial modalities. Many researchers agree that these modalities are complementary to each other and that the combination of the two can improve the accuracy in affective user models. However, there is a shortage of research in empirical work concerning the strengths and weaknesses of each modality so that more accurate recognizers can be built. In our research, we have investigated the recognition of emotions from the above mentioned modalities with respect to 6 basic emotional states, namely happiness,sadness, surprise, angerand disgustas well as the emotionless state which we refer to as neutral. We have found that certain states such as neutral happiness and surprise are more clearly recognized from the visual-facial modality whereas sadness and disgust are more clearly recognized from the audio-lingual modality.

[1]  Tsutomu Miyasato,et al.  Multimodal human emotion/expression recognition , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[2]  Zhigang Deng,et al.  Analysis of emotion recognition using facial expressions, speech and multimodal information , 2004, ICMI '04.

[3]  J. Russell,et al.  The psychology of facial expression: Frontmatter , 1997 .

[4]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[5]  Magda B. Arnold,et al.  The nature of emotion , 1968 .

[6]  P. Ekman,et al.  The nature of emotion: Fundamental questions. , 1994 .

[7]  Aaron Sloman,et al.  Fundamental Questions , 2006, KI.

[8]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[9]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[10]  P. Ekman.  Emotion in the human face , 1982 .

[11]  P. Ekman.  Unmasking The Face , 1975 .

[12]  Maria Virvou,et al.  Affective Student Modeling Based on Microphone and Keyboard User Actions , 2006 .

[13]  Rosalind W. Picard.  Affective computing: challenges , 2003, Int. J. Hum. Comput. Stud..

[14]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[15]  Volker Strom,et al.  Visual prosody: facial movements accompanying speech , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[16]  J. Russell.  Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. , 1994, Psychological bulletin.

[17]  P. Ekman.  An argument for basic emotions , 1992 .