Exploring the Body and Head Kinematics of Laughter, Filled Pauses and Breaths

We present ongoing work in the DUEL project, which focuses on the study of disfluencies, exclamations, and laughter in dialogue. Here we focus on the multimodal aspects of disfluent vocalizations, namely laughter and laughed speech, filled pauses, and breathing noises. We exemplify these phenomena in the rich multimodal Dream Apartment Corpus, a natural dialogue corpus, which, in addition to comprehensive disfluency and laughter annotation, comprises tracking data for the body and head. We discuss possible directions for developing models that can perceive as well as generate such multimodal behaviour.

[1]  Radoslaw Niewiadomski,et al.  Rhythmic Body Movements of Laughter , 2014, ICMI.

[2]  Eric Horvitz,et al.  Managing Human-Robot Engagement with Forecasts and... um... Hesitations , 2014, ICMI.

[3]  H. H. Clark,et al.  Using uh and um in spontaneous speaking , 2002, Cognition.

[4]  Junji Yamato,et al.  Analysis of Respiration for Prediction of "Who Will Be Next Speaker and When?" in Multi-Party Meetings , 2014, ICMI.

[5]  Alessandro Vinciarelli,et al.  Automatic Detection of Laughter and Fillers in Spontaneous Mobile Phone Conversations , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[6]  Günther Palm,et al.  Multimodal Laughter Detection in Natural Discourses , 2009, Human Centered Robot Systems, Cognition, Interaction, Technology.

[7]  Maurizio Mancini,et al.  Towards Automated Full Body Detection of Laughter Driven by Human Expert Annotation , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[8]  Matthew Purver,et al.  Strongly Incremental Repair Detection , 2014, EMNLP.

[9]  Bayya Yegnanarayana,et al.  Analysis of laughter and speech-laugh signals using excitation source information , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  Maurizio Mancini,et al.  Computing and Evaluating the Body Laughter Index , 2012, HBU.

[11]  Stefanos Zafeiriou,et al.  Audiovisual classification of vocal outbursts in human conversation using Long-Short-Term Memory networks , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[12]  Thierry Dutoit,et al.  AVLaughterCycle : Enabling a virtual agent to join in laughing with a conversational partner using a similarity-driven audiovisual laughter animation (Original Paper) , 2010 .

[13]  Petra Wagner,et al.  Exploring annotation of head gesture forms in spontaneous human interaction. , 2013 .

[14]  David Schlangen,et al.  MINT.tools: tools and adaptors supporting acquisition, annotation and analysis of multimodal corpora , 2013, INTERSPEECH.

[15]  William Curran,et al.  Laughter Type Recognition from Whole Body Motion , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.