A Conversational Agent that Reacts to Vocal Signals

Conversational agents are increasingly being used for training of social skills. One of their most important benefits is their ability to provide natural interaction with humans. This work proposes to extend conversational agents’ benefits for social skills training by analysing the emotion conveyed by the user’s speech. For that, we developed a new system that captures emotions from human voice and, combined with the context of a particular situation, uses this to influence the internal state of the agent and change its behaviour. An example of the system’s use is shown and its limitations and advantages are discussed, together with the internal workflow of the system.

[1]  Jaime C. Acosta,et al.  Achieving rapport with turn-by-turn, user-responsive emotional coloring , 2011, Speech Commun..

[2]  Catherine Pelachaud,et al.  Towards a Socially Adaptive Virtual Agent , 2015, IVA.

[3]  Albino Nogueiras,et al.  Speech emotion recognition using hidden Markov models , 2001, INTERSPEECH.

[4]  Kallirroi Georgila,et al.  SimSensei kiosk: a virtual human interviewer for healthcare decision support , 2014, AAMAS.

[5]  Marc Cavazza,et al.  Emotional input for character-based interactive storytelling , 2009, AAMAS.

[6]  Elisabetta Bevacqua,et al.  Multimodal Backchannels for Embodied Conversational Agents , 2010, IVA.

[7]  J. Russell,et al.  A 12-Point Circumplex Structure of Core Affect. , 2011, Emotion.

[8]  Björn W. Schuller,et al.  OpenEAR — Introducing the munich open-source emotion and affect recognition toolkit , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[9]  Frederik Van Broeckhoven,et al.  deLearyous : training interpersonal communication skills using unconstrained text input , 2012 .

[10]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[11]  Paula J. Durlach,et al.  BiLAT: A Game-Based Environment for Practicing Negotiation in a Cultural Context , 2009, Int. J. Artif. Intell. Educ..

[12]  K. Scherer,et al.  Appraisal processes in emotion: Theory, methods, research. , 2001 .

[13]  Tibor Bosse,et al.  Towards Aggression De-escalation Training with Virtual Agents: A Computational Model , 2014, HCI.

[14]  P. Ekman.  An argument for basic emotions , 1992 .

[15]  Johan Jeuring,et al.  Communicate! - A Serious Game for Communication Skills - , 2015, EC-TEL.

[16]  M. Theune,et al.  Social Behaviour in Police Interviews: Relating Data to Theories , 2015 .

[17]  Matthew Jensen Hays,et al.  Can Role-Play with Virtual Humans Teach Interpersonal Skills? , 2012 .

[18]  K. Scherer,et al.  Vocal expression of affect , 2005 .

[19]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[20]  Benjamin Lok,et al.  Audio Analysis of Human/Virtual-Human Interaction , 2008, IVA.

[21]  David A. van Leeuwen,et al.  Speech-based recognition of self-reported and observed emotion in a dimensional space , 2012, Speech Commun..

[22]  M. Yik.  A circumplex model of affect and its relation to personality : a five-language study , 1999 .

[23]  Léon J. M. Rothkrantz,et al.  Aggression Detection in Speech Using Sensor and Semantic Information , 2012, TSD.