Temporal, structural, and pragmatic synchrony between intonation and gesture

This paper explores the interaction between intonation and gesture, noting temporal, structural, and pragmatic synchrony. Videotapes of subjects conversing freely were annotated for intonation according to ToBI (Beckman and Elam 1997), and for gesture according to Kendon (1980) and McNeill (1992). The time-stamped annotations were analyzed statistically, as well as visually in the Anvil tool (Kipp 2001), which allows time-aligned viewing of videos with their annotations. Alignments were investigated between three levels of intonational units and four levels of gestural units. The intonational units were, from smallest to largest, pitch a ccents, intermediate phrases, and intonational phrases. The gestural units, again from smallest to largest, were apices of strokes, gesture phases, gesture phrases, and gesture units. Of these possible combinations, two pairs aligned. Apices clearly aligned with pitch accents, and gesture phrases tended to align with intermediate phrases. The existence of intermediate phrases in English has been the subject of some debate (Ladd 2008), and this paper suggests that a probable gestural correlate to intermediate phrases provides independent evidence for their existence. In addition, intonation and gesture were found to perform simultaneous complementary pragmatic functions. This temporal, structural, and pragmatic synchrony between the two channels reinforces the claim that speech and gesture are two surface forms of the same underlying and emerging cognitive content.

[1]  Ray L. Birdwhistell,et al.  Introduction to kinesics : an annotation system for analysis of body motion and gesture , 1952 .

[2]  A. Scheflen.  THE SIGNIFICANCE OF POSTURE IN COMMUNICATION SYSTEMS. , 1964, Psychiatry.

[3]  W. S. Condon,et al.  A segmentation of behavior , 1967 .

[4]  Kenneth L. Pike,et al.  Language in relation to a unified theory of the structure of human behavior, 2nd rev. ed. , 1967 .

[5]  A. Scheflen.  Human communication: Behavioral programs and their integration in interaction , 1968 .

[6]  David Crystal,et al.  Prosodic Systems and Intonation in English , 1969 .

[7]  R. Birdwhistell.  Kinesics and context , 1970 .

[8]  S. Duncan,et al.  Some Signals and Rules for Taking Speaking Turns in Conversations , 1972 .

[9]  A. Kendon.  Some Relationships Between Body Motion and Speech , 1972 .

[10]  Donald W. Fiske,et al.  Face-to-face interaction: Research, methods, and theory , 1977 .

[11]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[12]  D. Bolinger.  INTONATION AND GESTURE , 1983 .

[13]  J. Pierrehumbert,et al.  The Meaning of Intonational Contours in the Interpretation of Discourse , 1990 .

[14]  D. McNeill.  Hand and Mind: What Gestures Reveal about Thought , 1992 .

[15]  K. Tuite.  The production of gesture , 1993 .

[16]  W. Chafe.  Discourse, Consciousness, and Time: The Flow and Displacement of Conscious Experience in Speaking and Writing , 1996 .

[17]  M. Nespor,et al.  Prosody in Israeli Sign Language , 1999 .

[18]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[19]  Rashid Ansari,et al.  Multimodal human discourse: gesture and speech , 2002, TCHI.

[20]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[21]  Wolff-Michael Roth,et al.  From action to discourse: The bridging function of gestures , 2002, Cognitive Systems Research.

[22]  Ipke Wachsmuth,et al.  Deixis in Multimodal Human Computer Interaction: An Interdisciplinary Approach , 2003, Gesture Workshop.

[23]  Scott K. Liddell.  Grammar, Gesture, and Meaning in American Sign Language , 2003 .

[24]  Sherman Wilcox,et al.  Cross-linguistic and historical data from signed languages , 2004 .

[25]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[26]  María Luisa Flecha-García.  Eyebrow raising in dialogue: discourse structure, utterance function, and pitch accents , 2006 .

[27]  Maria L. Flecha-Garcia.  Non-verbal Communication in Dialogue: Alignment between Eyebrow Raises and Pitch Accents in English , 2007 .

[28]  A. Hübler.  The Nonverbal Shift in Early Modern English Conversation , 2007 .

[29]  Diane Brentari,et al.  The relationship between early gestures and intonation , 2008 .

[30]  Gina-Anne Levow,et al.  Cross-cultural investigation of prosody in verbal feedback in interactional rapport , 2010, INTERSPEECH.

[31]  Gaëlle Ferré,et al.  Timing Relationships between Speech and Co-Verbal Gestures in Spontaneous French , 2010, LREC 2010.