Modeling facial signs of appraisal during interaction: impact on users' perception and behavior

Interactive virtual characters are expected to lead to an intuitive interaction via multiple communicative modalities such as the expression of emotions. Generating facial expressions that are consistent with the interaction context is a challenge. This paper presents our interactive facial animation system based on the Component Process Model, generating facial signs of appraisal during a real-time interactive game. We describe a study comparing our model to the categorical approach of facial animation of emotion. Participants interacted with a virtual character in three conditions: no expression of emotion, an expression of a categorical emotion, and expressions of sequential signs of appraisal. The character in the appraisal condition was reported as being more expressive than in the other two conditions and was reported as experiencing more mental states. In addition, using appraisal signs modified the way participants interacted with the character (participants played slower after some emotions were expressed by the agent, i.e. pride and sadness).

[1]  J. Dunn,et al.  Understanding mind and emotion: longitudinal associations with mental-state talk between young friends. , 1998, Developmental psychology.

[2]  H. Conte,et al.  Circumplex models of personality and emotions , 1997 .

[3]  Klaus R. Scherer,et al.  Lost Luggage: A Field Study of Emotion–Antecedent Appraisal , 1997 .

[4]  Crystal L. Hoyt,et al.  Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology , 2002 .

[5]  P. D. Loor,et al.  MASCARET: A Pedagogical Multi-Agent System for Virtual Environments for Training , 2004, Int. J. Distance Educ. Technol..

[6]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[7]  K. Scherer,et al.  Multimodal expression of emotion: affect programs or componential appraisal patterns? , 2007, Emotion.

[8]  Nicole C. Krämer,et al.  "It doesn't matter what you are!" Explaining social effects of agents and avatars , 2010, Comput. Hum. Behav..

[9]  G. Clark,et al.  Reference , 2008 .

[10]  P. Ekman.  Emotions Revealed : Understanding Faces and Feelings , 2003 .

[11]  Radoslaw Niewiadomski,et al.  Multimodal Complex Emotions: Gesture Expressivity and Blended Facial Expressions , 2006, Int. J. Humanoid Robotics.

[12]  Jean-Claude Martin,et al.  Impact of Expressive Wrinkles on Perception of a Virtual Character's Facial Expressions of Emotions , 2009, IVA.

[13]  Catherine Pelachaud,et al.  Modelling multimodal expression of emotion in a virtual agent , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[14]  Marco Paleari.  Psychologically grounded avatars expressions , 2006 .

[15]  P. Young,et al.  Emotion and personality , 1963 .

[16]  Christoph Bartneck,et al.  Special issue on subtle expressivity for characters and robots , 2005, Int. J. Hum. Comput. Stud..

[17]  Kostas Karpouzis,et al.  Towards modeling embodied conversational agent character profiles using appraisal theory predictions in expression synthesis , 2009, Applied Intelligence.

[18]  Céline Clavel,et al.  MARC: a framework that features emotion models for facial animation during human–computer interaction , 2013, Journal on Multimodal User Interfaces.

[19]  K. Scherer.  Appraisal considered as a process of multilevel sequential checking. , 2001 .

[20]  Ronan Querrec,et al.  MASCARET: pedagogical multi-agents systems for virtual environment for training , 2003, Proceedings. 2003 International Conference on Cyberworlds.

[21]  Jonathan Gratch,et al.  Expression of Emotions Using Wrinkles, Blushing, Sweating and Tears , 2009, IVA.

[22]  Stacy Marsella,et al.  EMA: A process model of appraisal dynamics , 2009, Cognitive Systems Research.

[23]  P. Salovey,et al.  Measuring emotional intelligence with the MSCEIT V2.0. , 2003, Emotion.

[24]  J. Turner.  Human Emotions: A Sociological Theory , 2007 .

[25]  S. Baron-Cohen,et al.  The Cambridge Mindreading (CAM) Face-Voice Battery: Testing Complex Emotion Recognition in Adults with and without Asperger Syndrome , 2006, Journal of autism and developmental disorders.

[26]  Christoph Bartneck,et al.  Subtle expressivity for characters and robots , 2003, CHI Extended Abstracts.

[27]  Scherer.  The component process model: a blueprint for a comprehensive computational model of emotion , 2010 .

[28]  Koray Balci,et al.  XfaceEd: authoring tool for embodied conversational agents , 2005, ICMI '05.

[29]  Klaus R. Scherer,et al.  On the Sequential Nature of Appraisal Processes: Indirect Evidence from a Recognition Task , 1999 .

[30]  Radoslaw Niewiadomski,et al.  Affect expression in ECAs: Application to politeness displays , 2010, Int. J. Hum. Comput. Stud..

[31]  James J. Blascovich,et al.  Immersive Virtual Environment Technology as a tool for Psychological Science , 2001 .

[32]  Kai Vogeley,et al.  "Artificial humans": Psychology and neuroscience perspectives on embodiment and nonverbal communication , 2010, Neural Networks.

[33]  J. Stainer,et al.  The Emotions , 1922, Nature.

[34]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[35]  Didier Grandjean,et al.  Unpacking the cognitive architecture of emotion processes. , 2008, Emotion.

[36]  Paul Pauli,et al.  Modulation of facial mimicry by attitudes , 2008 .

[37]  Craig A. Smith,et al.  The psychology of facial expression: A Componential Approach to the meaning of facial expressions , 1997 .

[38]  Richard S. Lazarus,et al.  Emotions and adaptation: Conceptual and empirical relations. , 1968 .

[39]  James J. Gross,et al.  Emotion Generation and Emotion Regulation: One or Two Depends on Your Point of View , 2011, Emotion review : journal of the International Society for Research on Emotion.

[40]  Kristina Höök,et al.  Evaluating affective interactions , 2007, Int. J. Hum. Comput. Stud..

[41]  Klaus R. Scherer,et al.  Facial expressions allow inference of both emotions and their components , 2008 .

[42]  Elsevier Sdol.  International Journal of Human-Computer Studies , 2009 .

[43]  Hyniewska,et al.  Expression of affects in Embodied Conversational Agents , 2010 .

[44]  Radoslaw Niewiadomski,et al.  Modeling Emotional Expressions as Sequences of Behaviors , 2009, IVA.

[45]  J. Russell.  Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. , 1994, Psychological bulletin.

[46]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .