The Relation of Speech and Gestures: Temporal Synchrony Follows Semantic Synchrony

The close relationship of speech and gestures becomes conspicuously obvious in the temporal coordination of both modalities. In this paper we investigate in how far temporal synchrony is affected by the semantic relationship of gestures and their lexical affiliates. The results showed that when both modalities redundantly express the same information, the gesture's onset is closer to that of the accompanying lexical affiliate than when gestures convey complementary information: the closer speech and gestures are related semantically, the closer is their temporal relation. This novel finding is discussed with respect to implications for the production process of speech and gestures.

[1]  Maurizio Gentilucci,et al.  Speech and gesture share the same communication system , 2006, Neuropsychologia.

[2]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[3]  J. D. Ruiter.  The production of gesture and speech , 2000 .

[4]  GEOFFREY BEATTIE,et al.  Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation , 1999 .

[5]  Stefan Kopp,et al.  Increasing the expressiveness of virtual agents: autonomous generation of speech and gesture for spatial description tasks , 2009, AAMAS.

[6]  Stefan Kopp,et al.  Multimodal Communication from Multimodal Thinking - towards an Integrated Model of Speech and Gesture Production , 2008, Int. J. Semantic Comput..

[7]  Hao Yan,et al.  Paired speech and gesture generation in embodied conversational agents , 2000 .

[8]  Stefan Kopp,et al.  Verbal or Visual? How Information is Distributed across Speech and Gesture in Spatial Dialog , 2006 .

[9]  Yihsiu Chen,et al.  Language and Gesture: Lexical gestures and lexical access: a process model , 2000 .

[10]  De Ruiter,et al.  Gesture and speech production , 1998 .

[11]  E. Schegloff.  Structures of Social Action: On some gestures' relation to talk , 1985 .

[12]  J. Donald Ragsdale,et al.  Distribution of Kinesic Hesitation Phenomena in Spontaneous Speech , 1982 .

[13]  R. Krauss,et al.  Gesture, Speech, and Lexical Access: The Role of Lexical Movements in Speech Production , 1996 .

[14]  Rainer Schnell,et al.  Statistik für Soziologen , 2009 .

[15]  Geoffrey Beattie,et al.  An experimental investigation of the role of different types of iconic gesture in communication: A semantic feature approach , 2003 .

[16]  Joanna Blake,et al.  Gestures accompanying speech in specifically language-impaired children and their timing with speech , 2008 .

[17]  De Ruiter,et al.  Postcards from the mind: The relationship between speech, imagistic gesture and thought , 2007 .

[18]  Karen J. Pine,et al.  A microgenetic analysis of the relationship between speech and gesture in children: Evidence for semantic and temporal asynchrony , 2007 .

[19]  Susan Duncan,et al.  Growth points in thinking-for-speaking , 1998 .

[20]  Brian Butterworth,et al.  Gesture and Silence as Indicators of Planning in Speech , 1978 .

[21]  Hannes Rieser,et al.  On Factoring Out a Gesture Typology from the Bielefeld Speech-and-Gesture-Alignment Corpus (SAGA) , 2009, Gesture Workshop.

[22]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[23]  Judith Holler,et al.  How iconic gestures and speech interact in the representation of meaning: Are both aspects really integral to the process? , 2003 .

[24]  R. Krauss,et al.  Word Familiarity Predicts Temporal Asynchrony of Hand Gestures and Speech , 2010 .

[25]  Judith Holler,et al.  A micro-analytic investigation of how iconic gestures and speech represent core semantic features in talk , 2002 .

[26]  Sotaro Kita,et al.  Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders , 1997, Gesture Workshop.

[27]  Judith Holler,et al.  The interaction of iconic gesture and speech , 2004 .

[28]  Kawai Chui,et al.  Temporal patterning of speech and iconic gestures in conversational discourse , 2005 .

[29]  Heather Shovelton,et al.  Mapping the Range of Information Contained in the Iconic Hand Gestures that Accompany Spontaneous Speech , 1999 .

[30]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .