Timing Relationships between Speech and Co-Verbal Gestures in Spontaneous French

Several studies have described the links between gesture and speech in terms of timing, most of them concentrating on the production of hand gestures during speech or during pauses (Beattie & Aboudan, 1994; Nobe, 2000). Other studies have focused on the anticipation, synchronization or delay of gestures regarding their co-occurrence with speech (Schegloff, 1984; McNeill, 1992, 2005; Kipp, 2003; Loehr, 2004; Chui, 2005; Kida & Faraco, 2008; Leonard and Cummins, 2009) and we would like to participate in the debate in the present paper. We studied the timing relationships between iconic gestures and their lexical affiliates (Kipp, Neff et al., 2001) in a corpus of French conversational speech involving 6 speakers and annotated both in Praat (Boersma & Weenink, 2009) and Anvil (Kipp, 2001). The timing relationships we observed concerned the position of the gesture stroke as compared to that of the lexical affiliate and the Intonation Phrase, as well as the position of the gesture Phrase as regards that of the Intonation Phrase. The main results show that although gesture and speech are co-occurring, gestures generally start before the related speech segment.

[1]  D. McNeill Gesture and Thought , 2005 .

[2]  Jean-Luc Schwartz,et al.  Does the number of syllables affect the finger pointing movement in a pointing-naming task? , 2008 .

[3]  E. Selkirk,et al.  Phase theory and prosodic spellout: The case of verbs , 2007 .

[4]  S. Nobe Language and Gesture: Where do most spontaneous representational gestures actually occur with respect to speech? , 2000 .

[5]  Philippe Blache,et al.  Creating and Exploiting Multimodal Annotated Corpora: The ToMA Project , 2009, Multimodal Corpora.

[6]  E. Schegloff Structures of Social Action: On some gestures' relation to talk , 1985 .

[7]  Michael Neff,et al.  An annotation scheme for conversational gestures: how to economically capture timing and form , 2007, Lang. Resour. Evaluation.

[8]  D. McNeill Hand and Mind: What Gestures Reveal about Thought , 1992 .

[9]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[10]  R. Espesser,et al.  Le CID - Corpus of Interactional Data. Annotation et exploitation multimodale de parole conversationnelle [The “Corpus of Interactional Data” (CID) - Multimodal annotation of conversational speech”] , 2008, ICON.

[11]  Kawai Chui,et al.  Temporal patterning of speech and iconic gestures in conversational discourse , 2005 .

[12]  Michael Kipp,et al.  Gesture generation by imitation: from human behavior to computer character animation , 2005 .

[13]  G. Beattie,et al.  Gestures, pauses and speech: An experimental investigation of the effects of changing social context on their precise temporal relationships , 2009 .

[14]  P. Boersma Praat : doing phonetics by computer (version 5.1.05) , 2009 .

[15]  Nick Campbell,et al.  Tools & Resources for Visualising Conversational-Speech Interaction , 2008, LREC.

[16]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[17]  Ipke Wachsmuth,et al.  Deictic object reference in task-oriented dialogue , 2006 .

[18]  Dirk Heylen,et al.  Multimodal Corpora - From Models of Natural Interaction to Systems and Applications , 2009, Multimodal Corpora.

[19]  Les pauses démarcatives déplacées en anglais spontané. Marquage kinésique et prosodique , 2002 .