Hand Motion Oscillatory Gestures and Multimodal Discourse Analysis

Gesture and speech are part of a single human language system. They are coexpressive and complementary channels in the act of speaking. Whereas speech carries the major load of symbolic presentation, gesture provides the imagistic content. Proceeding from the established cotemporality of gesture and speech, our work on oscillatory gestures and multimodal discourse is discussed. Our new techniques of analyzing hand gestures in the frequency domain are described. By tracking an individual's hands during a speech, hand motion trajectory signals are extracted from real video datasets. Our wavelet-based approach in gestural oscillation extraction is presented as frequency ridges in a frequency-time space. Wavelet ridges are extracted from responses of wavelet analysis. These wavelet ridges are employed to characterize frequency properties of hand motion trajectory signals. Hand motion oscillatory gestures can be extracted from these frequency properties. The potential of such computational cross-modal language analysis is motivated by performing a microanalysis of 2 video datasets. In the first dataset, a participant describes her living space to an interlocutor. In the second, a participant describes her action plan to an interlocutor. The ability of our algorithm to extract gestural oscillations is demonstrated, and the way that oscillatory gestures reveal portions of the discourse structure is shown.

[1]  Francis K. H. Quek,et al.  Oscillatory gestures and discourse , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[2]  Susan Duncan,et al.  Growth points in thinking-for-speaking , 1998 .

[3]  Christian Wöhler,et al.  Motion-based recognition of pedestrians , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[4]  David McNeill,et al.  Growth Points, Catchments, and Contexts , 2000 .

[5]  Hironobu Fujiyoshi,et al.  Real-time human motion analysis by image skeletonization , 1998, Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No.98EX201).

[6]  Aaron M. Plotnik,et al.  Quantification of cyclic motion of marine animals from computer vision , 2002, OCEANS '02 MTS/IEEE.

[7]  James W. Davis,et al.  Categorical representation and recognition of oscillatory motion patterns , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[8]  Francis K. H. Quek,et al.  Gestural Hand Motion Oscillation and Symmetries for Multimodal Discourse: Detection and Analysis , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[9]  Mary P. Harper,et al.  Gestural spatialization in natural discourse segmentation , 2002, INTERSPEECH.

[10]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[11]  S. Mallat.  A wavelet tour of signal processing , 1998 .

[12]  Steven M. Seitz,et al.  View-Invariant Analysis of Cyclic Motion , 1997, International Journal of Computer Vision.

[13]  Rashid Ansari,et al.  Multimodal human discourse: gesture and speech , 2002, TCHI.

[14]  Bruno Torrésani,et al.  Multiridge detection and time-frequency reconstruction , 1999, IEEE Trans. Signal Process..

[15]  Edward H. Adelson,et al.  Analyzing and recognizing walking figures in XYT , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[16]  E. Adelson,et al.  Analyzing gait with spatiotemporal surfaces , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[17]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[18]  D. M. Titterington,et al.  Ridge Finding from Noisy Data , 1992 .

[19]  Daniel E. Koditschek,et al.  Dynamic system representation of basic and nonlinear in parameters oscillatory motion gestures , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[20]  Mubarak Shah,et al.  Cyclic motion detection for motion based recognition , 1994, Pattern Recognit..

[21]  Francis Quek,et al.  A parallel algorithm for dynamic gesture tracking , 1999, Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).

[22]  Francis K. H. Quek,et al.  Gestural Origo and Loci-Transitions in Natural Discourse Segmentation , 2001 .

[23]  Ajit S. Bopardikar,et al.  Wavelet transforms - introduction to theory and applications , 1998 .

[24]  Randal C. Nelson,et al.  Detection and Recognition of Periodic, Nonrigid Motion , 1997, International Journal of Computer Vision.

[25]  Julia Hirschberg,et al.  Instructions for annotating discourse , 1995 .

[26]  A. Grossmann,et al.  Cycle-octave and related transforms in seismic signal analysis , 1984 .

[27]  Francis K. H. Quek,et al.  Vector Coherence Mapping: A Parallelizable Approach to Image Flow Computation , 1998, ACCV.

[28]  Francis K. H. Quek.  The Catchment Feature Model: A Device for Multimodal Fusion and a Bridge between Signal and Sense , 2004, EURASIP J. Adv. Signal Process..

[29]  Gilles Fauconnier,et al.  Mental Spaces: Aspects of Meaning Construction in Natural Language , 1985 .

[30]  William T. Freeman.  Computer vision for television and games , 1999, Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).

[31]  Francis K. H. Quek,et al.  Gestural trajectory symmetries and discourse segmentation , 2002, INTERSPEECH.

[32]  Francis K. H. Quek,et al.  Hand gesture symmetric behavior detection and analysis in natural conversation , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[33]  Daniel E. Koditschek,et al.  Dynamical system representation, generation, and recognition of basic oscillatory motion gestures , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[34]  Fang Liu,et al.  Finding periodicity in space and time , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[35]  Francis Quek,et al.  Gesture cues for conversational interaction in monocular video , 1999, Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).