A kinematic analysis of prosodic structure in speech and manual gestures

Two experiments examining the effects of prosodic structure on the kinematic properties of speech and manual gestures are presented. Experiment 1 investigated the effects of prosodic boundaries, stress and their interaction on manual, oral, and intonation gestures (their duration and coordination). Experiment 2 investigated the effects of different types of prominence (deaccented, narrow focus, broad focus, contrastive focus) on oral constriction, intonation and manual gestures (duration and coordination). We recorded speech audio, vocal tract gestures (using electromagnetic articulometry) and manual movements (using motion capture). To the best of our knowledge, this is the first study to examine kinematic properties of body movement and vocal tract gestures concurrently. Preliminary results focus on the effects of prosodic structure on gesture duration and show that 1) manual and oral gestures are longer phrase-initially than phrasemedially and 2) manual and oral gestures lengthen under phrase-level prominence. [Supported by NIH].

[1]  Jeffrey J Berry,et al.  Accuracy of the NDI wave speech research system. , 2011, Journal of speech, language, and hearing research : JSLHR.

[2]  D. McNeill.  Hand and Mind , 1995 .

[3]  W. Cooper,et al.  Acoustical aspects of contrastive stress in question-answer contexts. , 1985, The Journal of the Acoustical Society of America.

[4]  Taehong Cho,et al.  Articulatory and acoustic studies on domain-initial strengthening in Korean , 2001, J. Phonetics.

[5]  Louis Goldstein,et al.  The coordination of boundary tones and its interaction with prominence , 2014, J. Phonetics.

[6]  Colin W. Wightman,et al.  Segmental durations in the vicinity of prosodic phrase boundaries. , 1992, The Journal of the Acoustical Society of America.

[7]  Jean-Luc Schwartz,et al.  The speech focus position effect on jaw-finger coordination in a pointing task. , 2008, Journal of speech, language, and hearing research : JSLHR.

[8]  W E Cooper,et al.  Acoustical Characteristics of Sentential Focus: Narrow vs. Broad and Single vs. Dual Focus Environments , 1986, Language and speech.

[9]  Louis Goldstein,et al.  Towards an articulatory phonology , 1986, Phonology.

[10]  P. Keating,et al.  Articulatory strengthening at edges of prosodic domains. , 1997, The Journal of the Acoustical Society of America.

[11]  Louis Goldstein,et al.  Coupling of tone and constriction gestures in pitch accents , 2012 .

[12]  G. Ayers,et al.  Guidelines for ToBI labelling , 1994 .

[13]  Martine Grice,et al.  The effect of focus marking on supralaryngeal articulation - Is it mediated by accentuation? , 2014, J. Phonetics.

[14]  M. Swerts,et al.  The Effects of Visual Beats on Prosodic Prominence: Acoustic Analyses, Auditory Perception and Visual Perception. , 2007 .

[15]  De Ruiter,et al.  Gesture and speech production , 1998 .

[16]  Núria Esteve-Gibert,et al.  Prosodic structure shapes the temporal realization of intonation and manual gesture movements. , 2013, Journal of speech, language, and hearing research : JSLHR.

[17]  M. Beckman,et al.  The articulatory kinematics of final lengthening. , 1991, The Journal of the Acoustical Society of America.

[18]  Emiel Krahmer,et al.  Visual prosody of newsreaders: Effects of information structure, emotional content and intended audience on facial expressions , 2010, J. Phonetics.

[19]  Louis Goldstein,et al.  Prosody and movement in American sign language: A task-dynamics approach , 2010 .

[20]  Taehong Cho.  Prosodic strengthening and featural enhancement: evidence from acoustic and articulatory realizations of /a,i/ in English. , 2005, The Journal of the Acoustical Society of America.

[21]  Dani Byrd,et al.  Spatiotemporal coupling between speech and manual motor actions , 2014, J. Phonetics.

[22]  Dani Byrd,et al.  The elastic phrase: modeling the dynamics of boundary-adjacent lengthening , 2003, J. Phonetics.

[23]  Stefanie Shattuck-Hufnagel,et al.  THE TIMING OF SPEECH-ACCOMPANYING GESTURES WITH RESPECT TO PROSODY , 2004 .

[24]  K. D. de Jong.  The supraglottal articulation of prominence in English: linguistic stress as localized hyperarticulation. , 1995, The Journal of the Acoustical Society of America.