An Autonomous Cognitive Empathy Model Responsive to Users’ Facial Emotion Expressions

Successful social robot services depend on how robots can interact with users. The effective service can be obtained through smooth, engaged, and humanoid interactions in which robots react properly to a user’s affective state. This article proposes a novel Automatic Cognitive Empathy Model, ACEM, for humanoid robots to achieve longer and more engaged human-robot interactions (HRI) by considering humans’ emotions and replying to them appropriately. The proposed model continuously detects the affective states of a user based on facial expressions and generates desired, either parallel or reactive, empathic behaviors that are already adapted to the user’s personality. Users’ affective states are detected using a stacked autoencoder network that is trained and tested on the RAVDESS dataset. The overall proposed empathic model is verified throughout an experiment, where different emotions are triggered in participants and then empathic behaviors are applied based on proposed hypothesis. The results confirm the effectiveness of the proposed model in terms of related social and friendship concepts that participants perceived during interaction with the robot.

[1]  Elizabeth A. Crane,et al.  Motion Capture and Emotion: Affect Detection in Whole Body Movement , 2007, ACII.

[2]  Chu Kiong Loo,et al.  Empathic Interaction Using the Computational Emotion Model , 2015, 2015 IEEE Symposium Series on Computational Intelligence.

[3]  N. Eisenberg,et al.  The Origins and Social Significance of Empathy-Related Responding. A Review of Empathy and Moral Development: Implications for Caring and Justice by M. L. Hoffman , 2001 .

[4]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[5]  James C. Lester,et al.  Modeling parallel and reactive empathy in virtual agents: an inductive approach , 2008, AAMAS.

[6]  Bram Vanderborght,et al.  Can the Social robot Probo Help children with autism to Identify Situation-Based Emotions? A Series of Single Case Experiments , 2013, Int. J. Humanoid Robotics.

[7]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[8]  Yasuo Kuniyoshi,et al.  A cognitive developmental scenario of transitional motor primitives acquisition. , 2007 .

[9]  H. Eysenck.  The structure of human personality , 1953 .

[10]  Dov Zohar,et al.  Effects of energy availability on immediate and delayed emotional reactions to work events. , 2003, The Journal of applied psychology.

[11]  Bram Vanderborght,et al.  Generic method for generating blended gestures and affective functional behaviors for social robots , 2018, Auton. Robots.

[12]  Bram Vanderborght,et al.  Expressing Emotions with the Social Robot Probo , 2010, Int. J. Soc. Robotics.

[13]  Shigeki Sugano,et al.  Emotional Communication between Humans and the Autonomous Robot WAMOEBA-2 (Waseda Amoeba) Which has the Emotion Model , 2000 .

[14]  Ana Paiva,et al.  Creating Individual Agents through Personality Traits , 2010, IVA.

[15]  Mitsuru Ishizuka,et al.  THE EMPATHIC COMPANION: A CHARACTER-BASED INTERFACE THAT ADDRESSES USERS' AFFECTIVE STATES , 2005, Appl. Artif. Intell..

[16]  T. Singer,et al.  The empathic brain: how, when and why? , 2006, Trends in Cognitive Sciences.

[17]  Adriana Tapus,et al.  Socially Assistive Robots: The Link between Personality, Empathy, Physiological Signals, and Task Performance , 2008, AAAI Spring Symposium: Emotion, Personality, and Social Behavior.

[18]  Jodi Forlizzi,et al.  All robots are not created equal: the design and perception of humanoid robot heads , 2002, DIS '02.

[19]  Adriana Tapus,et al.  Prosody-based adaptive metaphoric head and arm gestures synthesis in human robot interaction , 2013, 2013 16th International Conference on Advanced Robotics (ICAR).

[20]  L. Pessoa.  The Cognitive-Emotional Brain: From Interactions to Integration , 2013 .

[21]  Mark T. Greenberg,et al.  The Early Development of Conceptual Perspective Taking: Distinguishing among Multiple Perspectives , 1976 .

[22]  Tom Ziemke,et al.  How to Build a Supervised Autonomous System for Robot-Enhanced Therapy for Children with Autism Spectrum Disorder , 2017, Paladyn J. Behav. Robotics.

[23]  Shuo Yang,et al.  An ensemble classification algorithm for convolutional neural network based on AdaBoost , 2017, 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS).

[24]  Ana Paiva,et al.  Using Empathy to Improve Human-Robot Relationships , 2010, HRPR.

[25]  Bram Vanderborght,et al.  Humanoid Robot Pepper at a Belgian Chocolate Shop , 2018, HRI.

[26]  S. R. Livingstone,et al.  The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English , 2018, PloS one.

[27]  Stephan Hamann,et al.  Mapping discrete and dimensional emotions onto the brain: controversies and consensus , 2012, Trends in Cognitive Sciences.

[28]  Bram Vanderborght,et al.  A Personalized and Platform-Independent Behavior Control System for Social Robots in Therapy: Development and Applications , 2019, IEEE Transactions on Cognitive and Developmental Systems.

[29]  B. Englis,et al.  Expectations of Cooperation and Competition and Their Effects on Observers' Vicarious Emotional Responses , 1989 .

[30]  L. R. Goldberg.  THE DEVELOPMENT OF MARKERS FOR THE BIG-FIVE FACTOR STRUCTURE , 1992 .

[31]  Hiroshi Nakajima,et al.  The functionality of human-machine collaboration systems - mind model and social behavior , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[32]  J. Aharon-Peretz,et al.  Two systems for empathy: a double dissociation between emotional and cognitive empathy in inferior frontal gyrus versus ventromedial prefrontal lesions. , 2009, Brain : a journal of neurology.

[33]  Lauren Wispé,et al.  History of the concept of empathy. , 1987 .

[34]  Catherine Pelachaud,et al.  Modelling multimodal expression of emotion in a virtual agent , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[35]  D. Byrne.  Attitudes and Attraction , 1969 .

[36]  Elisabeth André,et al.  Creation and Evaluation of emotion expression with body movement, sound and eye color for humanoid robots , 2011, 2011 RO-MAN.

[37]  Adriana Tapus,et al.  A model for synthesizing a combined verbal and nonverbal behavior based on personality traits in human-robot interaction , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[38]  J. Russell,et al.  The Measurement of Affect, Mood, and Emotion: A Guide for Health-Behavioral Research , 2013 .

[39]  Richard A. Fabes,et al.  Empathy: Conceptualization, measurement, and relation to prosocial behavior , 1990 .

[40]  S. Baron-Cohen,et al.  The Empathy Quotient: An Investigation of Adults with Asperger Syndrome or High Functioning Autism, and Normal Sex Differences , 2004, Journal of autism and developmental disorders.

[41]  Azam Bagheri,et al.  A Novel Model for Emotion Detection from Facial Muscles Activity , 2019, ROBOT.

[42]  Minoru Asada,et al.  Emulation and behavior understanding through shared values , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[43]  Christoph Bartneck,et al.  Expressive robots in education: varying the degree of social supportive behavior of a robotic tutor , 2010, CHI.

[44]  Stefano Ferilli,et al.  Towards an Empathic Social Robot for Ambient Assisted Living , 2015, ESSEM@AAMAS.

[45]  Yong Tao,et al.  Compound facial expressions of emotion , 2014, Proceedings of the National Academy of Sciences.

[46]  D. Lefeber,et al.  Using the social robot probo as a social story telling agent for children with ASD , 2012 .

[47]  James C. Lester,et al.  Learning empathy: a data-driven framework for modeling empathetic companion agents , 2006, AAMAS '06.

[48]  Ana Paiva,et al.  The influence of empathy in human-robot relations , 2013, Int. J. Hum. Comput. Stud..

[49]  W. Köhler.  The task of Gestalt psychology , 1969 .

[50]  Salvatore Maria Aglioti,et al.  The pain of a model in the personality of an onlooker: Influence of state-reactivity and personality traits on embodied empathy for pain , 2009, NeuroImage.

[51]  B. J. Fogg,et al.  Can computer personalities be human personalities? , 1995, Int. J. Hum. Comput. Stud..

[52]  L. Riek,et al.  Real-Time Empathy : Facial Mimicry on a Robot , 2008 .

[53]  Candace L. Sidner,et al.  Where to look: a study of human-robot engagement , 2004, IUI '04.

[54]  Mark H. Davis.  Measuring individual differences in empathy: Evidence for a multidimensional approach. , 1983 .

[55]  M. Asada.  Development of artificial empathy , 2015, Neuroscience Research.

[56]  Omar Mubin,et al.  Social Robots in Public Spaces: A Meta-review , 2018, ICSR.

[57]  M. Mendelson,et al.  Measuring friendship quality in late adolescents and young adults: McGill Friendship Questionnaires. , 1999 .

[58]  Yasuo Kuniyoshi,et al.  Early motor development from partially ordered neural-body dynamics: experiments with a cortico-spinal-musculo-skeletal model , 2006, Biological Cybernetics.

[59]  Rosemary Blieszner,et al.  Definitions of friendship in the third age: age, gender, and study location effects , 2000 .

[60]  Haizhou Li,et al.  Making Social Robots More Attractive: The Effects of Voice Pitch, Humor and Empathy , 2013, Int. J. Soc. Robotics.

[61]  C. L. Philip Chen,et al.  Adaptive least squares support vector machines filter for hand tremor canceling in microsurgery , 2011, Int. J. Mach. Learn. Cybern..

[62]  Koen V. Hindriks,et al.  Mood expression through parameterized functional behavior of robots , 2013, 2013 IEEE RO-MAN.

[63]  S. Krach,et al.  Dissociating Empathy From Perspective-Taking: Evidence From Intra- and Inter-Individual Differences Research , 2019, Frontiers in Psychiatry.

[64]  Riitta Parkkola,et al.  Is emotional contagion special? An fMRI study on neural systems for affective and cognitive empathy , 2008, NeuroImage.

[65]  Clifford Nass,et al.  Consistency of personality in interactive characters: verbal cues, non-verbal cues, and user characteristics , 2000, Int. J. Hum. Comput. Stud..

[66]  S. Stich,et al.  The Varieties of Off-Line Simulation , 1996 .

[67]  Clifford Nass,et al.  Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent , 2005, Int. J. Hum. Comput. Stud..

[68]  Ben J. A. Kröse,et al.  Assessing Acceptance of Assistive Social Agent Technology by Older Adults: the Almere Model , 2010, Int. J. Soc. Robotics.

[69]  Ana Paiva,et al.  Empathic Robots for Long-term Interaction , 2014, Int. J. Soc. Robotics.

[70]  Iain R. Murray,et al.  Toward the simulation of emotion in synthetic speech: a review of the literature on human vocal emotion. , 1993, The Journal of the Acoustical Society of America.

[71]  N. Eisenberg,et al.  Empathy and its development , 1987 .

[72]  P. Costa,et al.  Normal Personality Assessment in Clinical Practice: The NEO Personality Inventory. , 1992 .

[73]  Bram Vanderborght,et al.  A Survey of Expectations About the Role of Robots in Robot-Assisted Therapy for Children with ASD: Ethical Acceptability, Trust, Sociability, Appearance, and Attachment , 2015, Science and Engineering Ethics.

[74]  R. Redfield,et al.  Measurement of the false positive rate in a screening program for human immunodeficiency virus infections. , 1988, The New England journal of medicine.

[75]  Donald R. Brown,et al.  Reciprocal Interaction and Similarity of Personality Attributes , 1984 .

[76]  Louis-Philippe Morency,et al.  OpenFace 2.0: Facial Behavior Analysis Toolkit , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[77]  Catherine Milne,et al.  Understanding engagement: Science demonstrations and emotional energy , 2007 .

[78]  Lori Ann Vogt.  Empathy and Moral Development: Implications for caring and Justice. , 2003 .

[79]  J. Turner,et al.  Handbook of the sociology of emotions , 2006 .

[80]  Ana Paiva,et al.  Modelling Empathy in Social Robotic Companions , 2011, UMAP Workshops.

[81]  P. Valdez,et al.  Effects of color on emotions. , 1994, Journal of experimental psychology. General.

[82]  D. Feil-Seifer,et al.  Defining socially assistive robotics , 2005, 9th International Conference on Rehabilitation Robotics, 2005. ICORR 2005..

[83]  Masaki Ogino,et al.  Mapping Facial Expression to Internal States Based on Intuitive Parenting , 2007, J. Robotics Mechatronics.

[84]  Oliver T Wolf,et al.  Who Cares? Revisiting Empathy in Asperger Syndrome , 2007, Journal of autism and developmental disorders.

[85]  William P. Hampes.  The Relation Between Humor Styles and Empathy , 2010 .

[86]  Rodolphe Gelin,et al.  A Mass-Produced Sociable Humanoid Robot: Pepper: The First Machine of Its Kind , 2018, IEEE Robotics & Automation Magazine.

[87]  G. Colombetti.  From affect programs to dynamical discrete emotions , 2009 .