Co-expressivity of Speech and Gesture: Lessons for Models of Aligned Speech and Gesture Production

When people combine language and gesture to convey their intended information, both modalities are characterized by an intriguing degree of coherence and consistency. For developing an account how speech and gesture are aligned to each other, one question of major importance is how meaning is distributed across the two channels. In this paper, we start from recent empirical findings indicating a flexible interaction between both systems and show that psycholinguistic models of speech and gesture production in literature cannot account for this interplay equally well. Based on a discussion of these theories as well as current computational approaches, we point out conclusions as to how a production model must be designed in order to simulate aligned, human-like multimodal behavior in virtual expressive agents.