Gesture and speech are part of a single human language system. They are co-expressive and complementary channels in the act of speaking. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. Proceeding from the established cotemporality of gesture and speech, we discuss our work on oscillatory gestures and speech. We present our wavelet-based approach in gestural oscillation extraction as geodesic ridges in frequency-time space. We motivate the potential of such computational cross-modal language analysis by performing a micro analysis of a video dataset in which a subject describes her living space. We demonstrate the ability of our algorithm to extract gestural oscillations and show how oscillatory gestures reveal portions of the discourse structure.
[1]
Julia Hirschberg,et al.
Instructions for annotating discourse
,
1995
.
[2]
David McNeill,et al.
Growth Points, Catchments, and Contexts
,
2000
.
[3]
Mary P. Harper,et al.
Gestural spatialization in natural discourse segmentation
,
2002,
INTERSPEECH.
[4]
Gilles Fauconnier,et al.
Mental Spaces: Aspects of Meaning Construction in Natural Language
,
1985
.
[5]
Francis Quek,et al.
A parallel algorithm for dynamic gesture tracking
,
1999,
Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).
[6]
Rashid Ansari,et al.
Multimodal human discourse: gesture and speech
,
2002,
TCHI.