An experimental study of speech/gesture interactions and distance encoding

This paper explores the possible encoding of distance information in vocal and manual pointing and its relationship with the linguistic structure of deictic words, as well as speech/gesture cooperation within the process of deixis. Two experiments required participants to point at and/or name a close or distant target, with speech only, with gesture only, or with speech+gesture. Acoustic, articulatory, and manual data were recorded. We investigated the interaction between vocal and manual pointing, with respect to the distance to the target. There are two major findings. First, distance significantly affects both articulatory and manual pointing, since participants perform larger vocal and manual gestures to designate a more distant target. Second, modality influences both deictic speech and gesture, since pointing is more emphatic in unimodal use of either over bimodal use of both, to compensate for the loss of the other mode. These findings suggest that distance is encoded in both vocal and manual pointing. We also demonstrate that the correlates of distance encoding in the vocal modality can be related to the typology of deictic words. Finally, our data suggest a two-way interaction between speech and gesture, and support the hypothesis that these two modalities are cooperating within a single communication system.

[1]  M. Tomasello,et al.  Understanding and sharing intentions: The origins of cultural cognition , 2005, Behavioral and Brain Sciences.

[2]  Philippe Maurer,et al.  Distance contrasts in demonstratives , 2013 .

[3]  Maurizio Gentilucci,et al.  Speech and gesture share the same communication system , 2006, Neuropsychologia.

[4]  Ursula Lenker,et al.  Ilse Wischer, Gabriele Diewald (Hrsg.): New Reflections on Grammaticalization. Amsterdam : John Benjamins, 2002. (Typological Studies in Language ; 49) , 2005 .

[5]  Willem J. M. Levelt,et al.  Pointing and voicing in deictic expressions , 1985 .

[6]  S. Kita,et al.  Demonstratives in Space and Interaction: Data from Lao Speakers and Implications for Semantic Analysis , 2003 .

[7]  Susan Duncan,et al.  Growth points in thinking-for-speaking , 1998 .

[8]  David A. Leavens,et al.  Understanding the Point of Chimpanzee Pointing , 2005, Current directions in psychological science.

[9]  S. Maeda.  On articulatory and acoustic variabilities , 1991 .

[10]  NANCY L. WOODWORTH,et al.  Sound symbolism in proximal and distal forms , 1991 .

[11]  Anne Cutler,et al.  A theory of lexical access in speech production , 1999, Behavioral and Brain Sciences.

[12]  W. Levelt,et al.  Speaking: From Intention to Articulation , 1990 .

[13]  S. Kita.  Pointing: Where language, culture, and cognition meet , 2003 .

[14]  H. Traunmüller.  Some aspects of the sound of speech sounds , 1987 .

[15]  D. McNeill.  Gesture and Thought , 2005 .

[16]  Jean-Luc Schwartz,et al.  Does the number of syllables affect the finger pointing movement in a pointing-naming task? , 2008 .

[17]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[18]  Yihsiu Chen,et al.  Language and Gesture: Lexical gestures and lexical access: a process model , 2000 .

[19]  M. Arbib.  From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics , 2005, Behavioral and Brain Sciences.

[20]  James Lubker,et al.  Formant frequencies of some fixed-mandible vowels and a model of speech motor programming by predictive simulation , 1979 .

[21]  Michael Tomasello,et al.  Gesture and the emergence and development of language , 2004 .

[22]  Sotaro Kita,et al.  What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking , 2003 .

[23]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[24]  M. Corbetta,et al.  Functional Organization of Human Intraparietal and Frontal Cortex for Attending, Looking, and Pointing , 2003, The Journal of Neuroscience.

[25]  Edith A. Moravcsik,et al.  Universals of human language , 1978 .

[26]  M. E. H. Schouten,et al.  The psychophysics of speech perception , 1987 .

[27]  Hartmut Traunmüller.  Sound symbolism in deictic words , 1996 .

[28]  Maurizio Gentilucci,et al.  Deictic word and gesture production: Their interaction , 2009, Behavioural Brain Research.

[29]  B. Butterworth,et al.  Gesture, speech, and computational stages: a reply to McNeill. , 1989, Psychological review.

[30]  G. Rizzolatti,et al.  Influence of different types of grasping on the transport component of prehension movements , 1991, Neuropsychologia.

[31]  Pierre Feyereisen,et al.  The competition between gesture and speech production in dual-task paradigms , 1997 .

[32]  M. Krause,et al.  Chimpanzee (Pan troglodytes) pointing: hand shapes, accuracy, and the role of eye gaze. , 1997, Journal of comparative psychology.

[33]  M. Swerts,et al.  The Effects of Visual Beats on Prosodic Prominence: Acoustic Analyses, Auditory Perception and Visual Perception. , 2007 .

[34]  De Ruiter,et al.  Gesture and speech production , 1998 .

[35]  E. Sapir.  A study in phonetic symbolism. , 1929 .

[36]  Massimo Vescovi,et al.  A kinematic approach to the conceptual representations of this and that , 2009, Cognition.

[37]  Christian Abry,et al.  The left inferior frontal gyrus under focus: an fMRI study of the production of deixis via syntactic extraction and prosodic focus , 2005, Journal of Neurolinguistics.

[38]  Autumn B. Hostetter,et al.  Visible embodiment: Gestures as simulated action , 2008, Psychonomic bulletin & review.

[39]  Susan Goldin-Meadow,et al.  Gesture Paves the Way for Language Development , 2005, Psychological science.