Visuo-auditory Multimodal Emotional Structure to Improve Human-Robot-Interaction

We propose an approach to analyze and synthesize a set of human facial and vocal expressions, and then use the classified expressions to decide the robot’s response in a human-robot-interaction. During a human-to-human conversation, a person senses the interlocutor’s face and voice, perceives her/his emotional expressions, and processes this information in order to decide which response to give. Moreover, observed emotions are taken into account and the response may be aggressive, funny (henceforth meaning humorous) or just neutral according to not only the observed emotions, but also the personality of the person. The purpose of our proposed structure is to endow robots with the capability to model human emotions, and thus several subproblems need to be solved: feature extraction, classification, decision and synthesis. In the proposed approach we integrate two classifiers for emotion recognition from audio and video, and then use a new method for fusion with the social behavior profile. To keep the person engaged in the interaction, after each iterance of analysis, the robot synthesizes human voice with both lips synchronization and facial expressions. The social behavior profile conducts the personality of the robot. The structure and work flow of the synthesis and decision are addressed, and the Bayesian networks are discussed. We also studied how to analyze and synthesize the emotion from the facial expression and vocal expression. A new probabilistic structure that enables a higher level of interaction between a human and a robot is proposed.

[1]  Carlo Strapparava,et al.  Getting serious about the development of computational humor , 2003, IJCAI 2003.

[2]  Nicu Sebe,et al.  Facial expression recognition from video sequences , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[3]  Application of Quantum Darwinism to Cosmic Inflation: An Example of the Limits Imposed in Aristotelian Logic by Information-based Approach to Gödel’s Incompleteness , 2010 .

[4]  C. Darwin,et al.  The Expression of the Emotions in Man and Animals , 1956 .

[5]  M. Milanova,et al.  Recognition of Emotional states in Natural Human-Computer Interaction , 2008, 2008 IEEE International Symposium on Signal Processing and Information Technology.

[6]  Hatice Gunes,et al.  Audio-Visual Classification and Fusion of Spontaneous Affective Data in Likelihood Space , 2010, 2010 20th International Conference on Pattern Recognition.

[7]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[8]  Marc Schröder,et al.  The SEMAINE API: Towards a Standards-Based Framework for Building Emotion-Oriented Systems , 2010, Adv. Hum. Comput. Interact..

[9]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[10]  Anoop K. Sinha,et al.  Suede: a Wizard of Oz prototyping tool for speech user interfaces , 2000, UIST '00.

[11]  A. Damasio.  Self comes to mind : constructing the conscious brain , 2010 .

[12]  Maja Pantic,et al.  Facial Expression Recognition , 2009, Encyclopedia of Biometrics.

[13]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[14]  G. Chaitin.  Meta Math!: The Quest for Omega , 2004, math/0404335.

[15]  Gayatri Suhas Paknikar,et al.  Facial Image Based Expression Classification System Using Committee Neural Networks , 2008 .

[16]  Constantine Kotropoulos,et al.  Emotional speech recognition: Resources, features, and methods , 2006, Speech Commun..

[17]  Carlo Strapparava,et al.  THE ACT OF CREATING HUMOROUS ACRONYMS , 2005, Appl. Artif. Intell..

[18]  Xiaoyi Feng.  Facial expression recognition based on local binary patterns and coarse-to-fine classification , 2004, The Fourth International Conference onComputer and Information Technology, 2004. CIT '04..

[19]  Jorge Dias,et al.  Horopter based dynamic background segmentation applied to an interactive mobile robot , 2009, 2009 International Conference on Advanced Robotics.

[20]  Sébastien George,et al.  An Approach to Automatic Analysis of Learners' Social Behavior During Computer-Mediated Synchronous Conversations , 2002, Intelligent Tutoring Systems.

[21]  Arne Jönsson,et al.  Wizard of Oz studies: why and how , 1993, IUI '93.

[22]  Acuerdo Para Entrevistanarrador.  Oh! , 1908 .

[23]  Graeme Ritchie,et al.  Prospects for Computational Humour , 2007 .

[24]  Stacy Marsella,et al.  Modeling the cognitive antecedents and consequences of emotion , 2009, Cognitive Systems Research.

[25]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[26]  P. Ekman,et al.  Facial Action Coding System: Manual , 1978 .

[27]  Narendra Ahuja,et al.  Detecting Faces in Images: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[29]  Ann Frederick,et al.  Waking the Tiger: Healing Trauma , 1997 .

[30]  Ling Guan,et al.  Recognizing human emotion from audiovisual information , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[31]  A. Damasio.  The feeling of what happens , 2001 .

[32]  Cynthia Breazeal,et al.  A Robotic Weight Loss Coach , 2007, AAAI.

[33]  Kim Binsted,et al.  Children's evaluation of computer-generated punning riddles , 1997 .

[34]  Nicu Sebe,et al.  Emotion recognition using a Cauchy Naive Bayes classifier , 2002, Object recognition supported by user interaction for service robots.

[35]  M. Sondhi,et al.  New methods of pitch extraction , 1968 .

[36]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[38]  W. Kaufmann,et al.  Social behavior profile in young males with fragile X syndrome: Characteristics and specificity , 2004, American journal of medical genetics. Part A.

[39]  Shrikanth S. Narayanan,et al.  Classifying emotions in human-machine spoken dialogs , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.