The Composite Sensing of Affect

This paper describes some of the issues faced by typical emotion recognition systems and the need to be able to deal with emotions in a natural setting. Studies tend to ignore the dynamic, versatile and personalised nature of affective expression and the influence that social setting, context and culture have on its rules of display. Affective cues can be present in multiple modalities and they can manifest themselves in different temporal order. Thus, fusing the feature sets is challenging. We present a composite approach to affective sensing. The term composite is used to reflect the blending of information from multiple modalities with the available semantic evidence to enhance the emotion recognition process.

[1]  Shrikanth Narayanan,et al.  Recognition of negative emotions from the speech signal , 2001, IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU '01..

[2]  K. Izdebski.  The Physics of Speech , 1980 .

[3]  P. Ekman.  Emotion in the human face , 1982 .

[4]  Stefanos D. Kollias,et al.  An intelligent system for facial emotion recognition , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[5]  Dilek Z. Hakkani-Tür,et al.  Using context to improve emotion detection in spoken dialog systems , 2005, INTERSPEECH.

[6]  Robert D. Ward,et al.  Affective computing: problems, reactions and intentions , 2004, Interact. Comput..

[7]  J. G. Taylor,et al.  Emotion recognition in human-computer interaction , 2005, Neural Networks.

[8]  Florian Schiel,et al.  The SmartKom Multimodal Corpus at BAS , 2002, LREC.

[9]  Iain R. Murray,et al.  Toward the simulation of emotion in synthetic speech: a review of the literature on human vocal emotion. , 1993, The Journal of the Acoustical Society of America.

[10]  Gang Wei,et al.  Speech emotion recognition based on HMM and SVM , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[11]  B. S. Manjunath,et al.  Introduction to MPEG-7: Multimedia Content Description Interface , 2002 .

[12]  Roddy Cowie,et al.  Beyond emotion archetypes: Databases for emotion modelling using neural networks , 2005, Neural Networks.

[13]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[14]  Björn W. Schuller,et al.  What Should a Generic Emotion Markup Language Be Able to Represent? , 2007, ACII.

[15]  Allen R. Hanson,et al.  Computer Vision Systems , 1978 .

[16]  L.C. De Silva,et al.  Real-time facial feature extraction and emotion recognition , 2003, Fourth International Conference on Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia. Proceedings of the 2003 Joint.

[17]  R. Gibson,et al.  What the Face Reveals , 2002 .

[18]  Valery A. Petrushin,et al.  EMOTION IN SPEECH: RECOGNITION AND APPLICATION TO CALL CENTERS , 1999 .

[19]  Del Moral HernandezEmilio.  2005 Special Issue , 2005 .

[20]  Rosalind W. Picard,et al.  Classical and novel discriminant features for affect recognition from speech , 2005, INTERSPEECH.

[21]  L. Devillers,et al.  F0 and pause features analysis for Anger and Fear detection in real-life spoken dialogs , 2004 .

[22]  FragopanagosN.,et al.  2005 Special Issue , 2005 .

[23]  Mark S. Nixon,et al.  Feature Extraction and Image Processing , 2002 .

[24]  Ing-Marie Jonsson,et al.  Automatic recognition of affective cues in the speech of car drivers to allow appropriate responses , 2005, OZCHI.

[25]  Cynthia Breazeal,et al.  Emotion and sociable humanoid robots , 2003, Int. J. Hum. Comput. Stud..

[26]  Steven J. Simske,et al.  Recognition of emotions in interactive voice response systems , 2003, INTERSPEECH.

[27]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[28]  Farshad Fotouhi,et al.  Using MPEG-7 to build a human brain image database for image-guided neurosurgery , 2005, SPIE Medical Imaging.

[29]  John R. Smith,et al.  MPEG-7 multimedia description schemes , 2001, IEEE Trans. Circuits Syst. Video Technol..

[30]  E. Velten.  A laboratory task for induction of mood states. , 1968, Behaviour research and therapy.

[31]  Russell Beale,et al.  Affect and Emotion in Human-Computer Interaction, From Theory to Applications , 2008, Affect and Emotion in Human-Computer Interaction.

[32]  P. Ekman.  Darwin, Deception, and Facial Expression , 2003, Annals of the New York Academy of Sciences.

[33]  Roddy Cowie,et al.  Describing the emotional states that are expressed in speech , 2003, Speech Commun..

[34]  Ing-Marie Jonsson,et al.  Using Paralinguistic Cues in Speech to Recognise Emotions in Older Car Drivers , 2008, Affect and Emotion in Human-Computer Interaction.

[35]  Roddy Cowie,et al.  ASR for emotional speech: Clarifying the issues and enhancing performance , 2005, Neural Networks.

[36]  Rana el Kaliouby,et al.  VERSUS PROSTHETIC TECHNOLOGIES IN AUTISM THERAPUETIC VERSUS PROSTHETIC ASSISTIVE TECHNOLOGIES : THE CASE OF AUTISM , 2005 .

[37]  R Stibbard.  Vocal expressions of emotions in non-laboratory speech : an investigation of the Reading/Leeds Emotion in Speech Project annotation data. , 2001 .

[38]  Frank Dellaert,et al.  Recognizing emotion in speech , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[39]  Lori Lamel,et al.  Challenges in real-life emotion annotation and machine learning based detection , 2005, Neural Networks.

[40]  David Sinclair,et al.  A Self-Referential Perceptual Inference Framework for Video Interpretation , 2003, ICVS.

[41]  David McNeill,et al.  Gesture and language dialectic , 2002 .

[42]  James Orwell,et al.  On the Use of MPEG-7 for Visual Surveillance , 2006 .

[43]  DevillersLaurence,et al.  2005 Special Issue , 2005 .

[44]  Jean-Claude Martin,et al.  Representing Real-Life Emotions in Audiovisual Data with Non Basic Emotional Patterns and Context Features , 2005, ACII.

[45]  Victor W. Zue.  Automatic speech recognition and understanding , 1990 .

[46]  Takeo Kanade,et al.  Automated facial expression recognition based on FACS action units , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[47]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[48]  Richard B. Reilly,et al.  Voice Pathology Assessment Based on a Dialogue System and Speech Analysis , 2004, AAAI Technical Report.

[49]  Klaus R. Scherer,et al.  Vocal communication of emotion: A review of research paradigms , 2003, Speech Commun..

[50]  Sumi Shigeno.  Cultural similarities and differences in the recognition of audio-visual speech stimuli , 1998, ICSLP.

[51]  Hirotaka Suzuki,et al.  Prosodic parameters in emotional speech , 1998, ICSLP.

[52]  Michael Wagner,et al.  Aspects of speaking-face data corpus design methodology , 2004, INTERSPEECH.

[53]  P. Robinson,et al.  The emotional hearing aid: an assistive tool for children with Asperger syndrome , 2005, Universal Access in the Information Society.

[54]  Loïc Kessous,et al.  Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech , 2008, Affect and Emotion in Human-Computer Interaction.

[55]  P. Ekman.  Unmasking The Face , 1975 .

[56]  T. S. Polzin,et al.  Verbal and non-verbal cues in the communication of emotions , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).