The Perception of Emotion in Artificial Agents

Given recent technological developments in robotics, artificial intelligence, and virtual reality, it is perhaps unsurprising that the arrival of emotionally expressive and reactive artificial agents is imminent. However, if such agents are to become integrated into our social milieu, it is imperative to establish an understanding of whether and how humans perceive emotion in artificial agents. In this review, we incorporate recent findings from social robotics, virtual reality, psychology, and neuroscience to examine how people recognize and respond to emotions displayed by artificial agents. First, we review how people perceive emotions expressed by an artificial agent, such as facial and bodily expressions. Second, we evaluate the similarities and differences in the consequences of perceived emotions in artificial compared to human agents. Besides accurately recognizing the emotional state of an artificial agent, it is critical to understand how humans respond to those emotions. Does interacting with an angry robot induce the same responses in people as interacting with an angry person? Similarly, does watching a robot rejoice when it wins a game elicit similar feelings of elation in the human observer? Here, we provide an overview of the current state of emotion expression and perception during interactions with artificial agents, as well as a clear articulation of the challenges and guiding principles to be addressed as we move ever closer to truly emotional artificial agents.

[1]  Christoph Bartneck,et al.  How Convincing is Mr. Data's Smile: Affective Expressions of Machines , 2001, User Modeling and User-Adapted Interaction.

[2]  Hiroshi Ishiguro,et al.  Evaluating facial displays of emotion for the android robot Geminoid F , 2011, 2011 IEEE Workshop on Affective Computational Intelligence (WACI).

[3]  F. Eyssel,et al.  Social categorization of social robots: anthropomorphism as a function of robot group membership. , 2012, The British journal of social psychology.

[4]  Ana Paiva,et al.  Using Empathy to Improve Human-Robot Relationships , 2010, HRPR.

[5]  Fabio Tesser,et al.  Interpretation of Emotional Body Language Displayed by a Humanoid Robot: A Case Study with Children , 2013, Int. J. Soc. Robotics.

[6]  Paul Pauli,et al.  Early cortical processing of natural and artificial emotional faces differs between lower and higher socially anxious persons , 2009, Journal of Neural Transmission.

[7]  Clara D. Martin,et al.  Controlling for interstimulus perceptual variance abolishes N170 face selectivity , 2007, Nature Neuroscience.

[8]  N. Kanwisher,et al.  Perceiving social interactions in the posterior superior temporal sulcus , 2017, Proceedings of the National Academy of Sciences.

[9]  Peter Mende-Siedlecki,et al.  Social attributions from faces: determinants, consequences, accuracy, and functional significance. , 2015, Annual review of psychology.

[10]  Mel Slater,et al.  Human Neuroscience , 2022 .

[11]  Mathieu Vandenbulcke,et al.  Affective scenes influence fear perception of individual body expressions , 2014, Human brain mapping.

[12]  Atsuo Takanishi,et al.  Design and control of 9-DOFs emotion expression humanoid arm , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[13]  Manfred Tscheligi,et al.  “I Love This Dog”—Children’s Emotional Attachment to the Robotic Dog AIBO , 2009, Int. J. Soc. Robotics.

[14]  Tatsuya Nomura,et al.  Comparison on Identification of Affective Body Motions by Robots Between Elder People and University Students: A Case Study in Japan , 2010, Int. J. Soc. Robotics.

[15]  Fabri,et al.  Expressive Agents : Non-verbal Communication in Collaborative Virtual Environments , 2002 .

[16]  J. Decety,et al.  Empathy as a driver of prosocial behaviour: highly conserved neurobehavioural mechanisms across species , 2016, Philosophical Transactions of the Royal Society B: Biological Sciences.

[17]  Mar Rus-Calafell,et al.  Creation of a new set of dynamic virtual reality faces for the assessment and training of facial emotion recognition ability , 2014, Virtual Reality.

[18]  T. Kanda,et al.  Measurement of negative attitudes toward robots , 2006 .

[19]  T. Allison,et al.  Social perception from visual cues: role of the STS region , 2000, Trends in Cognitive Sciences.

[20]  Rebecca Q. Stafford,et al.  Robots with Display Screens: A Robot with a More Humanlike Face Display Is Perceived To Have More Mind and a Better Personality , 2013, PloS one.

[21]  M. Bartlett,et al.  Bridging the Mechanical and the Human Mind: Spontaneous Mimicry of a Physically Present Android , 2014, PloS one.

[22]  Nicole C. Krämer,et al.  Investigations on empathy towards humans and robots using fMRI , 2014, Comput. Hum. Behav..

[23]  K. Scherer,et al.  Studying the dynamics of emotional expression using synthesized facial muscle movements. , 2000, Journal of personality and social psychology.

[24]  B. Gelder.  Towards the neurobiology of emotional body language , 2006, Nature Reviews Neuroscience.

[25]  Albert A. Rizzo,et al.  Performance-Driven Facial Animation: Basic Research on Human Judgments of Emotional State in Facial Avatars , 2001, Cyberpsychology Behav. Soc. Netw..

[26]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[27]  Adrian Hilton,et al.  A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling , 2011, 2011 International Conference on Computer Vision.

[28]  Emiliano Ricciardi,et al.  Distinct Neural Systems Involved in Agency and Animacy Detection , 2011, Journal of Cognitive Neuroscience.

[29]  Hillary Anger Elfenbein,et al.  Reading your Counterpart: The Benefit of Emotion Recognition Accuracy for Effectiveness in Negotiation , 2006 .

[30]  Alexander Todorov,et al.  The amygdala and FFA track both social and non-social face dimensions , 2010, Neuropsychologia.

[31]  Ravi Vaidyanathan,et al.  Design and testing of a hybrid expressive face for a humanoid robot , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[32]  Elisabeth André,et al.  Creation and Evaluation of emotion expression with body movement, sound and eye color for humanoid robots , 2011, 2011 RO-MAN.

[33]  Brett Stevens,et al.  Emotional body language displayed by artificial agents , 2012, TIIS.

[34]  Y. Trope,et al.  Body Cues, Not Facial Expressions, Discriminate Between Intense Positive and Negative Emotions , 2012, Science.

[35]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[36]  Oliver G. B. Garrod,et al.  Facial expressions of emotion are not culturally universal , 2012, Proceedings of the National Academy of Sciences.

[37]  Paul Pauli,et al.  Electromyographic responses to static and dynamic avatar emotional facial expressions. , 2006, Psychophysiology.

[38]  Sophie Jörg,et al.  Evaluating the emotional content of human motions on real and virtual characters , 2008, APGV '08.

[39]  C. Nass,et al.  Are Machines Gender Neutral? Gender‐Stereotypic Responses to Computers With Voices , 1997 .

[40]  James Everett Young,et al.  Poor Thing! Would You Feel Sorry for a Simulated Robot? A comparison of empathy toward a physical and a simulated robot , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[41]  Aaron Powers,et al.  Matching robot appearance and behavior to tasks to improve human-robot cooperation , 2003, The 12th IEEE International Workshop on Robot and Human Interactive Communication, 2003. Proceedings. ROMAN 2003..

[42]  Clare Press,et al.  Neuroscience and Biobehavioral Reviews Action Observation and Robotic Agents: Learning and Anthropomorphism , 2022 .

[43]  R. Adolphs,et al.  Emotion Perception from Face, Voice, and Touch: Comparisons and Convergence , 2017, Trends in Cognitive Sciences.

[44]  Beatrice de Gelder,et al.  The Body as a Tool for Anger Awareness—Differential Effects of Angry Facial and Bodily Expressions on Suppression from Awareness , 2015, PloS one.

[45]  N. Moray,et al.  Trust in automation. Part II. Experimental studies of trust and human intervention in a process control simulation. , 1996, Ergonomics.

[46]  Stefan Kopp,et al.  To Err is Human(-like): Effects of Robot Gesture on Perceived Anthropomorphism and Likability , 2013, International Journal of Social Robotics.

[47]  Candace L. Sidner,et al.  A design methodology for expressing emotion on robot faces , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[48]  A. Takanishi,et al.  Various emotional expressions with emotion expression humanoid robot WE-4RII , 2004, IEEE Conference on Robotics and Automation, 2004. TExCRA Technical Exhibition Based..

[49]  Alois Knoll,et al.  Biological movement increases acceptance of humanoid robots as human partners in motor interaction , 2011, AI & SOCIETY.

[50]  Friederike Anne Eyssel,et al.  My robot is more human than yours: Effects of group membership on anthropomorphic judgments of social robots , 2011, IROS 2011.

[51]  Mark A. Neerincx,et al.  Facial and vocal emotion expression of a personal computer assistant to engage, educate and motivate children , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[52]  Emily S. Cross,et al.  The Control of Automatic Imitation Based on Bottom–Up and Top–Down Cues to Animacy: Insights from Brain and Behavior , 2014, Journal of Cognitive Neuroscience.

[53]  Jacqueline Nadel,et al.  Reading sadness beyond human faces , 2010, Brain Research.

[54]  Emily S. Cross,et al.  Robotic movement preferentially engages the action observation network , 2012, Human brain mapping.

[55]  E. Broadbent.  Interactions With Robots: The Truths We Reveal About Ourselves , 2017, Annual review of psychology.

[56]  Konstantinos Kafetsios,et al.  Signal and Noise in the Perception of Facial Emotion Expressions , 2016, Personality & social psychology bulletin.

[57]  K. Scherer,et al.  FACSGen: A Tool to Synthesize Emotional Facial Expressions Through Systematic Manipulation of Facial Action Units , 2011 .

[58]  T. Chaminade,et al.  Motor interference between Humans and Humanoid Robots: Effect of Biological and Artificial Motion , 2005, Proceedings. The 4nd International Conference on Development and Learning, 2005..

[59]  Ewald Moser,et al.  Amygdala activation at 3T in response to human and avatar facial expressions of emotions , 2007, Journal of Neuroscience Methods.

[60]  Sonya S. Kwak,et al.  What makes people empathize with an emotional robot?: The impact of agency and physical embodiment on human empathy for a robot , 2013, 2013 IEEE RO-MAN.

[61]  Brian T. Gill,et al.  "Robovie, you'll have to go into the closet now": children's social and moral relationships with a humanoid robot. , 2012, Developmental psychology.

[62]  S. Preston,et al.  Empathy: Its ultimate and proximate bases. , 2001, The Behavioral and brain sciences.

[63]  Brian Scassellati,et al.  The similarity-attraction effect in human-robot interaction , 2010, 2010 IEEE 9th International Conference on Development and Learning.

[64]  Tatsuya Nomura,et al.  Prediction of Human Behavior in Human--Robot Interaction Using Psychological Scales for Anxiety and Negative Attitudes Toward Robots , 2008, IEEE Transactions on Robotics.

[65]  L. Cañamero,et al.  Interpretation of emotional body language displayed by robots , 2010, AFFINE '10.

[66]  Kai Vogeley,et al.  Why we interact: On the functional role of the striatum in the subjective experience of social interaction , 2014, NeuroImage.

[67]  Maria V. Sanchez-Vives,et al.  A Virtual Reprise of the Stanley Milgram Obedience Experiments , 2006, PloS one.

[68]  Kerstin Dautenhahn,et al.  Socially intelligent robots: dimensions of human–robot interaction , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[69]  Rachael E. Jack,et al.  Culture Shapes How We Look at Faces , 2008, PloS one.

[70]  Oliver G. B. Garrod,et al.  Realistic facial animation generation based on facial expression mapping , 2014, International Conference on Graphic and Image Processing.

[71]  Friederike Eyssel,et al.  When a Robot’s Group Membership Matters , 2013, International Journal of Social Robotics.

[72]  Aude Billard,et al.  Humanoid robots versus humans: How is emotional valence of facial expressions recognized by individuals with schizophrenia? An exploratory study , 2016, Schizophrenia Research.

[73]  Andrea Kleinsmith,et al.  Cross-cultural differences in recognizing affect from body posture , 2006, Interact. Comput..

[74]  Friederike Eyssel,et al.  Minimal Group - Maximal Effect? Evaluation and Anthropomorphization of the Humanoid Robot NAO , 2011, ICSR.

[75]  Christoph Bartneck,et al.  Meta analysis of the usage of the Godspeed Questionnaire Series , 2015, 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[76]  Sonya S. Kwak,et al.  Evaluation of the expressions of robotic emotions of the emotional robot, "Mung" , 2009, ICUIMC '09.

[77]  Cynthia Breazeal,et al.  Emotion and sociable humanoid robots , 2003, Int. J. Hum. Comput. Stud..

[78]  Nicole C. Krämer,et al.  An Experimental Study on Emotional Reactions Towards a Robot , 2012, International Journal of Social Robotics.

[79]  C. Darwin.  The Expression of Emotion in Man and Animals , 2020 .

[80]  B. de Gelder,et al.  The many faces of the emotional body , 2014 .

[81]  Margot J. Taylor,et al.  N170 or N1? Spatiotemporal differences between object and face processing using ERPs. , 2004, Cerebral cortex.

[82]  K. Vogeley,et al.  Toward a second-person neuroscience 1 , 2013, Behavioral and Brain Sciences.

[83]  Patrik Vuilleumier,et al.  Self-relevance processing in the human amygdala: gaze direction, facial expression, and emotion intensity. , 2009, Emotion.

[84]  Lisa Feldman Barrett,et al.  Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture. , 2014, Emotion.

[85]  C. Bartneck,et al.  In your face, robot! The influence of a character's embodiment on how users perceive its emotional expressions , 2004 .

[86]  Alexis Hervais-Adelman,et al.  Amygdala Activation for Eye Contact Despite Complete Cortical Blindness , 2013, The Journal of Neuroscience.

[87]  Alexander Todorov,et al.  Evaluating face trustworthiness: a model based approach. , 2008, Social cognitive and affective neuroscience.

[88]  J. Haxby,et al.  fMRI Responses to Video and Point-Light Displays of Moving Humans and Manipulable Objects , 2003, Journal of Cognitive Neuroscience.

[89]  P. Downing,et al.  The role of occipitotemporal body-selective regions in person perception , 2011, Cognitive neuroscience.

[90]  Alois Knoll,et al.  Moving Just Like You: Motor Interference Depends on Similar Motility of Agent and Observer , 2012, PloS one.

[91]  Jamil Zaki,et al.  PSYCHOLOGICAL SCIENCE Research Article It Takes Two The Interpersonal Nature of Empathic Accuracy , 2022 .

[92]  Bruce A. MacDonald,et al.  The Effects of Synthesized Voice Accents on User Perceptions of Robots , 2011, Int. J. Soc. Robotics.

[93]  Mikko Sams,et al.  The effect of dynamics on identifying basic emotions from synthetic and natural faces , 2008, Int. J. Hum. Comput. Stud..

[94]  J. Nadel,et al.  Human brain spots emotion in non humanoid robots. , 2011, Social cognitive and affective neuroscience.

[95]  P. Ekman,et al.  What is Meant by Calling Emotions Basic , 2011 .

[96]  Rachael E. Jack,et al.  Internal representations reveal cultural diversity in expectations of facial expressions of emotion. , 2012, Journal of experimental psychology. General.

[97]  Rachael E. Jack,et al.  Toward a Social Psychophysics of Face Communication , 2017, Annual review of psychology.

[98]  Jekaterina Novikova,et al.  A design model of emotional body expressions in non-humanoid robots , 2014, HAI.

[99]  Karl F. MacDorman,et al.  The Uncanny Valley [From the Field] , 2012, IEEE Robotics Autom. Mag..

[100]  Susanne Quadflieg,et al.  In our own image? Emotional and neural processing differences when observing human–human vs human–robot interactions , 2015, Social cognitive and affective neuroscience.

[101]  P. Niedenthal,et al.  Social functionality of human emotion. , 2012, Annual review of psychology.

[102]  E. Harmon-Jones,et al.  Empathy and attitudes: can feeling for a member of a stigmatized group improve feelings toward the group? , 1997, Journal of personality and social psychology.

[103]  Takayuki Kanda,et al.  Escaping from Children’s Abuse of Social Robots , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[104]  B. de Gelder.  Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. , 2009, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[105]  Ravi Vaidyanathan,et al.  Assessment of human response to robot facial expressions through visual evoked potentials , 2010, 2010 10th IEEE-RAS International Conference on Humanoid Robots.

[106]  C. Izard.  Innate and universal facial expressions: evidence from developmental and cross-cultural research. , 1994, Psychological bulletin.

[107]  Patrice Renaud,et al.  Virtual Faces Expressing Emotions: An Initial Concomitant and Construct Validity Study , 2014, Front. Hum. Neurosci..

[108]  S. Blakemore,et al.  Atypical interference effect of action observation in autism spectrum conditions , 2013, Psychological Medicine.

[109]  Jun Hu,et al.  Exploring the abuse of robots , 2008 .

[110]  Tobias Brosch,et al.  Faces in Context: A Review and Systematization of Contextual Influences on Affective Face Processing , 2012, Front. Psychology.

[111]  Jenay M. Beer,et al.  Recognizing Emotion in Virtual Agent, Synthetic Human, and Human Facial Expressions , 2010 .

[112]  J. Nadel,et al.  Human responses to an expressive robot , 2006 .

[113]  Y. Paulignan,et al.  An Interference Effect of Observed Biological Movement on Action , 2003, Current Biology.

[114]  Anthony G. Pipe,et al.  Perception of own and robot engagement in human-robot interactions and their dependence on robotics knowledge , 2014, Robotics Auton. Syst..

[115]  Matthias Rehm,et al.  Experimental designs for cross-cultural interactions: A case study on affective body movements for HRI , 2012, 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012).

[116]  Aryel Beck,et al.  Towards an Affect Space for robots to display emotional body language , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[117]  Kerstin Eder,et al.  Believing in BERT: Using expressive communication to enhance trust and counteract operational error in physical Human-robot interaction , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[118]  Francesco Ferrari,et al.  Blurring Human–Machine Distinctions: Anthropomorphic Appearance in Social Robots as a Threat to Human Distinctiveness , 2016, International Journal of Social Robotics.

[119]  Klaus Mathiak,et al.  Recognition Profile of Emotions in Natural and Virtual Faces , 2008, PloS one.

[120]  Friederike Eyssel,et al.  ‘If you sound like me, you must be more human’: On the interplay of robot and user features on human-robot acceptance and anthropomorphism , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[121]  Tony Belpaeme,et al.  People Interpret Robotic Non-linguistic Utterances Categorically , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[122]  Aude Billard,et al.  Assessing Interaction Dynamics in the Context of Robot Programming by Demonstration , 2013, International Journal of Social Robotics.

[123]  Marco Tamietto,et al.  Standing up for the body. Recent progress in uncovering the networks involved in the perception of bodies and bodily expressions , 2010, Neuroscience & Biobehavioral Reviews.

[124]  Emily S. Cross,et al.  The shaping of social perception by stimulus and knowledge cues to human animacy , 2016, Philosophical Transactions of the Royal Society B: Biological Sciences.

[125]  James Townsend,et al.  Making faces: Creating three-dimensional parameterized models of facial expression , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[126]  A. Young,et al.  Emotion Perception from Dynamic and Static Body Expressions in Point-Light and Full-Light Displays , 2004, Perception.

[127]  Candace L. Sidner,et al.  Explorations in engagement for humans and robots , 2005, Artif. Intell..

[128]  Jamy Li,et al.  The benefit of being physically present: A survey of experimental works comparing copresent robots, telepresent robots and virtual agents , 2015, Int. J. Hum. Comput. Stud..

[129]  Mikko Sams,et al.  Identification of synthetic and natural emotional facial expressions , 2003, AVSP.

[130]  Mariano Alcañiz Raya,et al.  Affective Interactions Using Virtual Reality: The Link between Presence and Emotions , 2007, Cyberpsychology Behav. Soc. Netw..

[131]  Claudia Wagner,et al.  Anthropomorphic inferences from emotional nonverbal cues: A case study , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[132]  Lola Cañamero,et al.  Are Discrete Emotions Useful in Human-Robot Interaction? Feedback from Motion Capture Analysis , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[133]  T. Heatherton,et al.  Detecting agency from the biological motion of veridical vs animated agents. , 2007, Social cognitive and affective neuroscience.

[134]  Paolo Dario,et al.  How safe are service robots in urban environments? Bullying a robot , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[135]  Karon E. MacLean,et al.  The Role of Affective Touch in Human-Robot Interaction: Human Intent and Expectations in Touching the Haptic Creature , 2012, Int. J. Soc. Robotics.

[136]  Dong-Soo Kwon,et al.  Realization of expressive body motion using leg-wheel hybrid mobile robot: KaMERo1 , 2008, 2008 International Conference on Control, Automation and Systems.

[137]  Michiteru Kitazaki,et al.  Measuring empathy for human and robot hand pain using electroencephalography , 2015, Scientific Reports.

[138]  Scott S. Snibbe,et al.  Experiences with Sparky, a Social Robot , 2002 .

[139]  S. Milgram.  BEHAVIORAL STUDY OF OBEDIENCE. , 1963, Journal of abnormal psychology.

[140]  D. De Rossi,et al.  Can a Humanoid Face be Expressive? A Psychophysiological Investigation , 2015, Front. Bioeng. Biotechnol..

[141]  G. Rizzolatti,et al.  Brain Response to a Humanoid Robot in Areas Implicated in the Perception of Human Emotional Gestures , 2010, PloS one.

[142]  U. Hess,et al.  Modulation of facial reactions to avatar emotional faces by nonconscious competition priming. , 2009, Psychophysiology.

[143]  Wan Ling Chang,et al.  The effect of group size on people's attitudes and cooperative behaviors toward robots in interactive gameplay , 2012, 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.