Analyzing Interpersonal Empathy via Collective Impressions

This paper presents a research framework for understanding the empathy that arises between people while they are conversing. By focusing on the process by which empathy is perceived by other people, this paper aims to develop a computational model that automatically infers perceived empathy from participant behavior. To describe such perceived empathy objectively, we introduce the idea of using the collective impressions of external observers. In particular, we focus on the fact that the perception of other's empathy varies from person to person, and take the standpoint that this individual difference itself is an essential attribute of human communication for building, for example, successful human relationships and consensus. This paper describes a probabilistic model of the process that we built based on the Bayesian network, and that relates the empathy perceived by observers to how the gaze and facial expressions of participants co-occur between a pair. In this model, the probability distribution represents the diversity of observers' impression, which reflects the individual differences in the schema when perceiving others' empathy from their behaviors, and the ambiguity of the behaviors. Comprehensive experiments demonstrate that the inferred distributions are similar to those made by observers.

[1]  Carlos Busso,et al.  IEMOCAP: interactive emotional dyadic motion capture database , 2008, Lang. Resour. Evaluation.

[2]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[3]  William Ickes,et al.  Naturalistic Social Cognition: Empathic Accuracy in Mixed-Sex Dyads , 1990 .

[4]  Klaus R. Scherer,et al.  Using Actor Portrayals to Systematically Study Multimodal Emotion Expression: The GEMEP Corpus , 2007, ACII.

[5]  Sigal G. Barsade The Ripple Effect: Emotional Contagion and its Influence on Group Behavior , 2002 .

[6]  Yoichi Sato,et al.  Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates , 2007, International Journal of Computer Vision.

[7]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[8]  S. Preston,et al.  Empathy: Its ultimate and proximate bases. , 2001, The Behavioral and brain sciences.

[9]  Javier R. Movellan,et al.  Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , 2009, NIPS.

[10]  Klaus R. Scherer,et al.  Vocal communication of emotion: A review of research paradigms , 2003, Speech Commun..

[11]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[12]  Masakiyo Fujimoto,et al.  A realtime multimodal system for analyzing group meetings by combining face pose tracking and speaker diarization , 2008, ICMI '08.

[13]  K. Scherer,et al.  The New Handbook of Methods in Nonverbal Behavior Research , 2008 .

[14]  Frank J. Bernieri,et al.  The Importance of Nonverbal Cues in Judging Rapport , 1999 .

[15]  Dennis Reidsma,et al.  Exploiting ‘Subjective’ Annotations , 2008, COLING 2008.

[16]  Daniel Gatica-Perez,et al.  Exploiting observers' judgements for nonverbal group interaction analysis , 2011, Face and Gesture 2011.

[17]  Apa Nlm,et al.  Modeling Continuous Self-Report Measures of Perceived Emotion Using Generalized Additive Mixed Models , 2013 .

[18]  W. James Potter,et al.  Rethinking validity and reliability in content analysis , 1999 .

[19]  P. Hekkert,et al.  The emotion slider: A self-report device for the continuous measurement of emotion , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[20]  Junji Yamato,et al.  Understanding communicative emotions from collective external observations , 2012, CHI EA '12.

[21]  Gary J McKeown,et al.  Modeling continuous self-report measures of perceived emotion using generalized additive mixed models. , 2014, Psychological methods.

[22]  P. Ekman,et al.  Coherence between expressive and experiential systems in emotion , 1994 .

[23]  E. Brunswik Perception and the Representative Design of Psychological Experiments , 1957 .

[24]  Kazuhiro Otsuka Conversation Scene Analysis [Social Sciences] , 2011, IEEE Signal Processing Magazine.

[25]  M. Knapp,et al.  Nonverbal communication in human interaction , 1972 .

[26]  A. Bouhuys,et al.  Induction of depressed and elated mood by music influences the perception of facial emotional expressions in healthy subjects. , 1995, Journal of affective disorders.

[27]  Daniel Gatica-Perez,et al.  The YouTube Lens: Crowdsourced Personality Impressions and Audiovisual Analysis of Vlogs , 2013, IEEE Transactions on Multimedia.

[28]  L. Tiedens,et al.  The Social Life of Emotions: THE INTERPERSONAL CONTEXT , 2004 .

[29]  Witold Pedrycz,et al.  Face recognition: A study in information fusion using fuzzy integral , 2005, Pattern Recognit. Lett..

[30]  Daniel Gatica-Perez,et al.  Analyzing Group Interactions in Conversations: a Review , 2006, 2006 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems.

[31]  T. Chartrand,et al.  The chameleon effect: the perception-behavior link and social interaction. , 1999, Journal of personality and social psychology.

[32]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[33]  R. Rosenthal Meta-analytic procedures for social research , 1984 .

[34]  Ted S. Sindlinger,et al.  Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business , 2010 .

[35]  Roddy Cowie,et al.  Describing the emotional states that are expressed in speech , 2003, Speech Commun..

[36]  M. Bartlett,et al.  Machine Analysis of Facial Expressions , 2007 .

[37]  S. Baron-Cohen,et al.  Does the autistic child have a “theory of mind” ? , 1985, Cognition.

[38]  Björn W. Schuller,et al.  AVEC 2011-The First International Audio/Visual Emotion Challenge , 2011, ACII.

[39]  R. Thayer The biopsychology of mood and arousal , 1989 .

[40]  Lori Lamel,et al.  Challenges in real-life emotion annotation and machine learning based detection , 2005, Neural Networks.

[41]  Andrea Kleinsmith,et al.  Multi-score Learning for Affect Recognition: The Case of Body Postures , 2011, ACII.

[42]  Junji Yamato,et al.  Analyzing empathetic interactions based on the probabilistic modeling of the co-occurrence patterns of facial expressions in group meetings , 2011, Face and Gesture 2011.

[43]  J. Russell,et al.  The psychology of facial expression: Frontmatter , 1997 .

[44]  R. Levenson,et al.  Empathy: a physiological substrate. , 1992, Journal of personality and social psychology.

[45]  Sung-Hyuk Cha Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions , 2007 .

[46]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[47]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Nicole Chovil Discourse‐oriented facial displays in conversation , 1991 .

[49]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[50]  P. Niedenthal Embodying Emotion , 2007, Science.

[51]  Mohammad Soleymani,et al.  Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus , 2010 .

[52]  Gerardo Hermosillo,et al.  Learning From Crowds , 2010, J. Mach. Learn. Res..

[53]  Wade Junek,et al.  Mind Reading: The Interactive Guide to Emotions , 2007 .

[54]  R. Levenson,et al.  Continuous measurement of emotion: The affect rating dial. , 2007 .

[55]  Mark H. Davis The Social Life of Emotions: Empathy , 2004 .

[56]  Sung-Hyuk Cha,et al.  On measuring the distance between histograms , 2002, Pattern Recognit..

[57]  Michael Vitale,et al.  The Wisdom of Crowds , 2015, Cell.

[58]  Joost Broekens,et al.  Real time labeling of affect in music using the affectbutton , 2010, AFFINE '10.

[59]  Maja Pantic,et al.  The first facial expression recognition and analysis challenge , 2011, Face and Gesture 2011.

[60]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[61]  Kazuhiro Otsuka,et al.  Conversation scene analysis based on dynamic Bayesian network and image-based gaze detection , 2010, ICMI-MLMI '10.

[62]  James A. Russell,et al.  Facing others: A social communicative perspective on facial displays , 1997 .

[63]  A. J. Conger Integration and generalization of kappas for multiple raters. , 1980 .

[64]  T. Singer,et al.  The Social Neuroscience of Empathy , 2009, Annals of the New York Academy of Sciences.

[65]  Maja Pantic,et al.  The SEMAINE corpus of emotionally coloured character interactions , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[66]  C. Batson These things called empathy: Eight related but distinct phenomena. , 2009 .

[67]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[68]  J. Bavelas,et al.  "I show how you feel": Motor mimicry as a communicative act. , 1986 .

[69]  Friedhelm Schwenker,et al.  Investigating fuzzy-input fuzzy-output support vector machines for robust voice quality classification , 2013, Comput. Speech Lang..

[70]  Elmar Nöth,et al.  "Of all things the measure is man" automatic classification of emotions and inter-labeler consistency [speech-based emotion recognition] , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[71]  S. Asch Opinions and Social Pressure , 1955, Nature.

[72]  Hatice Gunes,et al.  Output-associative RVM regression for dimensional and continuous emotion prediction , 2011, Face and Gesture 2011.

[73]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.