Functional and temporal relations between spoken and gestured components of language : A corpus-based inquiry

Based on the Bielefeld Speech and Gesture Alignment Corpus ( Lucking et al. 2013 ), this paper presents a systematic comparison of the linguistic characteristics of unimodal (speech only) and multimodal (gesture-accompanied) forms of language use. The results suggest that each of these two modes of expression is characterized by statistical preferences for certain types of words and grammatical categories. The words that are most frequently accompanied by a manual gesture, when controlled for their total frequency, include unspecific spatial lexemes, various deictic words, and particles that express difficulty in word retrieval or formulation. Other linguistic items, including pronouns and verbs of cognition, show a strong dispreference for being gesture-accompanied. The second part of the paper shows that gestures do not occur within a fixed time window relative to the word(s) they relate to, but the preferred temporal distance varies with the type of functional relation that exists between the verbal and gestural channel.

[1]  Stefan Kopp,et al.  The Relation of Speech and Gestures: Temporal Synchrony Follows Semantic Synchrony , 2011 .

[2]  Ronald Carter,et al.  Spoken Corpus Linguistics: From Monomodal to Multimodal , 2015 .

[3]  Fred J. Damerau,et al.  Generating and Evaluating Domain-Oriented Multi-Word Terms from Texts , 1993, Inf. Process. Manag..

[4]  Ellen Fricke.  Origo, Geste und Raum : Lokaldeixis im Deutschen , 2007 .

[5]  Hannes Rieser,et al.  On Factoring Out a Gesture Typology from the Bielefeld Speech-and-Gesture-Alignment Corpus (SAGA) , 2009, Gesture Workshop.

[6]  Stefan Kopp,et al.  Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications , 2013, Journal on Multimodal User Interfaces.

[7]  Kasper Kok,et al.  The grammatical potential of co-speech gesture: A Functional Discourse Grammar perspective , 2016 .

[8]  R. Langacker.  Foundations of Cognitive Grammar: Volume I: Theoretical Prerequisites , 1987 .

[9]  Jana Bressem,et al.  3. Gestures and speech from a linguistic perspective: A new field and its history , 2013 .

[10]  Ellen Fricke,et al.  Grammatik multimodal : wie Wörter und Gesten zusammenwirken , 2012 .

[11]  Dawn Knight,et al.  HeadTalk, HandTalk and the corpus: towards a framework for multi-modal, multi-media corpus development , 2009 .

[12]  Douglas G Altman,et al.  How to obtain the P value from a confidence interval , 2011, BMJ : British Medical Journal.

[13]  J. Streeck.  Gesture as communication I: Its coordination with gaze and speech , 1993 .

[14]  Maria Thurmair,et al.  Modalpartikeln und ihre Kombinationen , 1989 .

[15]  Stefan Kopp,et al.  Individualized Gesturing Outperforms Average Gesturing - Evaluating Gesture Production in Virtual Humans , 2010, IVA.

[16]  Jürgen Streeck,et al.  Grammars, Words, and Embodied Meanings: On the Uses and Evolution of So and Like , 2002 .

[17]  Eric Sanders,et al.  The IFADV Corpus: a Free Dialog Video Corpus , 2008, LREC.

[18]  Alan Cienki,et al.  Cognitive Grammar and gesture: Points of convergence, advances and challenges , 2016 .

[19]  Stefan Kopp,et al.  Multimodal Communication from Multimodal Thinking - towards an Integrated Model of Speech and Gesture Production , 2008, Int. J. Semantic Comput..

[20]  Alan Cienki,et al.  Usage events of spoken language and the symbolic units we (may) abstract from them , 2012 .

[21]  N. J. Enfield,et al.  On linear segmentation and combinatorics in co-speech gesture: A symmetry-dominance construction in Lao fish trap descriptions* , 2004 .

[22]  Ernest W.B. Hess-Lüttich.  Ellen Fricke, Origo, Geste und Raum, Lokaldeixis im Deutschen, Linguistik, Impulse und Tendenzen, 24, Berlin, Walter de Gruyter, 2007 [Rezension] , 2008 .

[23]  Stefan Diemer,et al.  Compiling computer-mediated spoken language corpora: key issues and recommendations , 2016 .

[24]  Florian Schiel,et al.  Signal processing via web services: The use case WebMAUS , 2012 .

[25]  Yihsiu Chen,et al.  Language and Gesture: Lexical gestures and lexical access: a process model , 2000 .

[26]  Stefan Kopp,et al.  Mapping out the multifunctionality of speakers’ gestures , 2016 .

[27]  Uri Hadar,et al.  Iconic gestures: the grammatical categories of lexical affiliates , 1999, Journal of Neurolinguistics.

[28]  A. Kendon.  Gestures as illocutionary and discourse structure markers in Southern Italian conversation , 1995 .

[29]  Fred Cummins,et al.  The temporal relation between beat gestures and speech , 2011 .

[30]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[31]  D. McNeill.  Hand and Mind: What Gestures Reveal about Thought , 1992 .

[32]  Erhard W. Hinrichs,et al.  WebLicht: Web-Based LRT Services for German , 2010, ACL.

[33]  Daniel Hole,et al.  Definite type and indefinite token: the article son in colloquial german , 2000 .

[34]  Dawn Knight.  Multimodality and Active Listenership: A Corpus Approach , 2011 .

[35]  Katya Saint-Amand,et al.  Alignment of Speech and Co-speech Gesture in a Constraint-based Grammar , 2013 .

[36]  Stefan Kopp,et al.  GNetIc - Using Bayesian Decision Networks for Iconic Gesture Generation , 2009, IVA.

[37]  David McNeill,et al.  Language and Gesture: Catchments and contexts: non-modular factors in speech and gesture production , 2000 .

[38]  Palmer Morrel-Samuels,et al.  Word familiarity predicts temporal asynchrony of hand gestures and speech. , 1992 .

[39]  Eli Hinkel,et al.  Ten Criteria for a Spoken Grammar , 2001 .

[40]  David McNeill,et al.  Speech, Gesture, and Discourse. , 1992 .

[41]  Steven Schoonjans,et al.  Modalpartikeln als multimodale Konstruktionen. Eine korpusbasierte Kookkurrenzanalyse von Modalpartikeln und Gestik im Deutschen. , 2018 .