Speech and manual gesture coordination in a pointing task

This study explores the coordination between manual pointing gestures and gestures of the vocal tract. Using a novel methodology that allows for concurrent collection of audio, kinematic body and speech articulator trajectories, we ask 1) which particular gesture (vowel gesture, consonant gesture, or tone gesture) the pointing gesture is coordinated with, and 2) with which landmarks the two gestures are coordinated (for example, whether the pointing gesture is coordinated to the speech gesture by the onset or maximum displacement). Preliminary results indicate coordination of the intonation gesture and the pointing gesture.

[1]  A. Kendon.  Some Relationships Between Body Motion and Speech , 1972 .

[2]  U. Hadar,et al.  Head Movement Correlates of Juncture and Stress at Sentence Level , 1983, Language and speech.

[3]  Betty Tuller,et al.  A “Dynamic Pattern” Perspective on the Control and Coordination of Movement , 1983 .

[4]  D. McNeill.  So you think gestures are nonverbal , 1985 .

[5]  J. Pierrehumbert,et al.  The Meaning of Intonational Contours in the Interpretation of Discourse , 1990 .

[6]  Evelyn McClave,et al.  Gestural beats: The rhythm hypothesis , 1994 .

[7]  G. Ayers,et al.  Guidelines for ToBI labelling , 1994 .

[8]  D. McNeill.  Hand and Mind , 1995 .

[9]  Alice Turk,et al.  The domain of accentual lengthening in American English , 1997 .

[10]  De Ruiter,et al.  Gesture and speech production , 1998 .

[11]  Alice Turk,et al.  A cross-linguistic study of accentual lengthening: Dutch vs. English , 1999 .

[12]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[13]  Jeffery A. Jones,et al.  Visual Prosody and Speech Intelligibility , 2004, Psychological science.

[14]  Stefanie Shattuck-Hufnagel,et al.  THE TIMING OF SPEECH-ACCOMPANYING GESTURES WITH RESPECT TO PROSODY , 2004 .

[15]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[16]  D. McNeill.  Gesture and Thought , 2005 .

[17]  Maurizio Gentilucci,et al.  Speech and gesture share the same communication system , 2006, Neuropsychologia.

[18]  L. Goldstein,et al.  Manifestation of prosodic structure in articulatory variation: Evidence from lip kinematics in English , 2006 .

[19]  Sotaro Kita,et al.  On-line Integration of Semantic Information from Speech and Gesture: Insights from Event-related Brain Potentials , 2007, Journal of Cognitive Neuroscience.

[20]  Roel M. Willems,et al.  When language meets action: the neural integration of gesture and speech. , 2007, Cerebral cortex.

[21]  Jean-Luc Schwartz,et al.  The speech focus position effect on jaw-finger coordination in a pointing task. , 2008, Journal of speech, language, and hearing research : JSLHR.

[22]  Man Gao,et al.  Mandarin tones : an articulatory phonology account , 2008 .

[23]  Marion Dohen,et al.  Co-production of contrastive prosodic focus and manual gestures: temporal coordination and effects on the acoustic and articulatory correlates of focus , 2010, Speech Prosody 2010.

[24]  Louis Goldstein,et al.  Prosody and movement in American sign language: A task-dynamics approach , 2010 .

[25]  Emiel Krahmer,et al.  Visual prosody of newsreaders: Effects of information structure, emotional content and intended audience on facial expressions , 2010, J. Phonetics.

[26]  Jeffrey J Berry,et al.  Accuracy of the NDI wave speech research system. , 2011, Journal of speech, language, and hearing research : JSLHR.

[27]  Fred Cummins,et al.  The temporal relation between beat gestures and speech , 2011 .

[28]  Norma C Mendoza-Denton,et al.  Semiotic Layering through Gesture and Intonation , 2011 .

[29]  Louis Goldstein,et al.  Coupling of tone and constriction gestures in pitch accents , 2012 .

[30]  Núria Esteve-Gibert,et al.  Prosodic structure shapes the temporal realization of intonation and manual gesture movements. , 2013, Journal of speech, language, and hearing research : JSLHR.

[31]  Guest Editorial.  Gesture and speech in interaction : An overview , 2013 .

[32]  Dani Byrd,et al.  Spatiotemporal coupling between speech and manual motor actions , 2014, J. Phonetics.

[33]  Mark K. Tiede,et al.  A kinematic analysis of prosodic structure in speech and manual gestures , 2015, ICPhS.