Analyzing Temporal Dynamics of Dyadic Synchrony in Affective Interactions

Human communication is a dynamical and interactive process that naturally induces an active flow of interpersonal coordination, and synchrony, along various behavioral dimensions. Assessing and characterizing the temporal dynamics of synchrony during an interaction is essential for fully understanding the human communication mechanisms. In this work, we focus on uncovering the temporal variability patterns of synchrony in visual gesture and vocal behavior in affectively rich interactions. We propose a statistical scheme to robustly quantify the turnwise interpersonal synchrony. The analysis of the synchrony dynamics measure relies heavily on functional data analysis techniques. Our analysis results reveal that: 1) the dynamical patterns of interpersonal synchrony differ depending on the global emotions of an interaction dyad; 2) there generally exists a tight dynamical emotion-synchrony coupling over the interaction. These observations corroborate that interpersonal behavioral synchrony is a critical manifestation of the underlying affective processes, shedding light toward improved affective interaction modeling and automatic emotion recognition.

[1]  S L Johnson,et al.  Sequential interactions in the marital communication of depressed men and women. , 2000, Journal of consulting and clinical psychology.

[2]  Sergey Levine,et al.  Real-time prosody-driven synthesis of body language , 2009, SIGGRAPH 2009.

[3]  Roddy Cowie,et al.  FEELTRACE: an instrument for recording perceived emotion in real time , 2000 .

[4]  David Haussler,et al.  Exploiting Generative Models in Discriminative Classifiers , 1998, NIPS.

[5]  Angeliki Metallinou,et al.  Analysis and Predictive Modeling of Body Language Behavior in Dyadic Interactions From Multimodal Interlocutor Cues , 2014, IEEE Transactions on Multimedia.

[6]  Pedro J. Moreno,et al.  Using the Fisher kernel method for Web audio classification , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[7]  Athanasios Katsamanis,et al.  Computing vocal entrainment: A signal-derived PCA-based quantification scheme with application to affect analysis in married couple interactions , 2014, Comput. Speech Lang..

[8]  B. Silverman,et al.  Canonical correlation analysis when the data are curves. , 1993 .

[9]  Shrikanth S. Narayanan,et al.  Modeling mutual influence of multimodal behavior in affective dyadic interactions , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  Shrikanth S. Narayanan,et al.  Modeling Dynamics of Expressive Body Gestures In Dyadic Interactions , 2017, IEEE Transactions on Affective Computing.

[11]  Gang Hua,et al.  Probabilistic Elastic Matching for Pose Variant Face Verification , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Ikuo Daibo,et al.  Interactional Synchrony in Conversations about Emotional Episodes: A Measurement by “the Between-Participants Pseudosynchrony Experimental Paradigm” , 2006 .

[13]  Douglas E. Sturim,et al.  Support vector machines using GMM supervectors for speaker verification , 2006, IEEE Signal Processing Letters.

[14]  C M Murphy,et al.  Couple communication patterns of maritally aggressive and nonaggressive male alcoholics. , 1997, Journal of studies on alcohol.

[15]  Shrikanth S. Narayanan,et al.  The effect of word frequency and lexical class on articulatory-acoustic coupling , 2013, INTERSPEECH.

[16]  Angeliki Metallinou,et al.  Annotation and processing of continuous emotional attributes: Challenges and opportunities , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[17]  Zhigang Deng,et al.  Analysis of emotion recognition using facial expressions, speech and multimodal information , 2004, ICMI '04.

[18]  T. Chartrand,et al.  The chameleon effect: the perception-behavior link and social interaction. , 1999, Journal of personality and social psychology.

[19]  Carlos Busso,et al.  The USC CreativeIT database of multimodal dyadic interactions: from speech and full body motion capture to continuous emotional annotations , 2015, Language Resources and Evaluation.

[20]  Michael J. Richardson,et al.  Measuring the Dynamics of Interactional Synchrony , 2012 .

[21]  B. Silverman,et al.  Functional Data Analysis , 1997 .

[22]  Florent Perronnin,et al.  Fisher Kernels on Visual Vocabularies for Image Categorization , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[24]  Albert Ali Salah,et al.  Fisher vectors with cascaded normalization for paralinguistic analysis , 2015, INTERSPEECH.

[25]  Julia Hirschberg,et al.  Measuring Acoustic-Prosodic Entrainment with Respect to Multiple Levels and Dimensions , 2011, INTERSPEECH.

[26]  A. Murat Tekalp,et al.  Analysis of Head Gesture and Prosody Patterns for Prosody-Driven Head-Gesture Animation , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.