Analysis of interaction attitudes using data-driven hand gesture phrases

Hand gesture is one of the most expressive, natural and common types of body language for conveying attitudes and emotions in human interactions. In this paper, we study the role of hand gesture in expressing attitudes of friendliness or conflict towards the interlocutors during interactions. We first employ an unsupervised clustering method using a parallel HMM structure to extract recurring patterns of hand gesture (hand gesture phrases or primitives). We further investigate the validity of the derived hand gesture phrases by examining the correlation of dyad's hand gesture for different interaction types defined by the attitudes of interlocutors. Finally, we model the interaction attitudes with SVM using the dynamics of the derived hand gesture phrases over an interaction. The classification results are promising, suggesting the expressiveness of the derived hand gesture phrases for conveying attitudes and emotions.

[1]  Angeliki Metallinou,et al.  Toward body language generation in dyadic interaction settings from interlocutor multimodal cues , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[2]  Jessica K. Hodgins,et al.  Hierarchical Aligned Cluster Analysis for Temporal Clustering of Human Motion , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Athanasios Katsamanis,et al.  Tracking continuous emotional trends of participants during affective dyadic interactions using body language and speech information , 2013, Image Vis. Comput..

[4]  Mohamed Chetouani,et al.  Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines , 2012, IEEE Transactions on Affective Computing.

[5]  Yael Edan,et al.  Vision-based hand-gesture applications , 2011, Commun. ACM.

[6]  Jure Leskovec,et al.  Predicting positive and negative links in online social networks , 2010, WWW '10.

[7]  Shrikanth Narayanan,et al.  The USC Creative IT Database: A Multimodal Database of Theatrical Improvisation , 2010 .

[8]  Sergey Levine,et al.  Real-time prosody-driven synthesis of body language , 2009, ACM Trans. Graph..

[9]  A. Murat Tekalp,et al.  Analysis of Head Gesture and Prosody Patterns for Prosody-Driven Head-Gesture Animation , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  K. Scherer,et al.  The New Handbook of Methods in Nonverbal Behavior Research , 2008 .

[11]  Peter Robinson,et al.  Detecting Affect from Non-stylised Body Motions , 2007, ACII.

[12]  Swapna Somasundaran,et al.  QA with Attitude: Exploiting Opinion Type Analysis for Improving Question Answering in On-line Discussions and the News , 2007, ICWSM.

[13]  I. Kimbara.  Interpersonal influences on gesture production : evidence for gesture form convergence across speakers in dyadic interaction , 2006 .

[14]  Peter F. Driessen,et al.  Gesture-Based Affective Computing on Motion Capture Data , 2005, ACII.

[15]  Zhigang Deng,et al.  Analysis of emotion recognition using facial expressions, speech and multimodal information , 2004, ICMI '04.

[16]  Jernej Barbic,et al.  Segmenting Motion Capture Data into Distinct Behaviors , 2004, Graphics Interface.

[17]  Andreas Stolcke,et al.  Prosody-based automatic detection of annoyance and frustration in human-computer dialog , 2002, INTERSPEECH.

[18]  Shrikanth Narayanan,et al.  Recognition of negative emotions from the speech signal , 2001, IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU '01..

[19]  Horst Bunke,et al.  On the influence of vocabulary size and language models in unconstrained handwritten text recognition , 2001, Proceedings of Sixth International Conference on Document Analysis and Recognition.

[20]  Sharon Marie Carnicke,et al.  Stanislavsky in Focus: An Acting Master for the Twenty-First Century , 1998 .

[21]  Steve Young,et al.  The HTK book , 1995 .

[22]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[23]  Kit Includes,et al.  [Installation guide]. , 1978, Le Chirurgien-dentiste de France.

[24]  P. Ekman,et al.  BODY POSITION, FACIAL EXPRESSION, AND VERBAL BEHAVIOR DURING INTERVIEWS. , 1964, Journal of abnormal psychology.