Gestural Hand Motion Oscillation and Symmetries for Multimodal Discourse: Detection and Analysis

We address the detection and analysis of gestural hand motion oscillation and symmetries in natural speech. First, we extract hand motion trajectory signals from video dataset. Second, we present our windowed correlation coefficient approach for gestural symmetry extraction. The signs and magnitudes of the correlation coefficients in the cardinal directions of the subject's torso characterize the symmetries. Third, we present a wavelet-based approach that extracts the time-frequency properties of hand motion oscillation. By analyzing these frequency properties durations of homogeneous gestural oscillations are detected. Finally, we apply our approach to a real video dataset captured in candid conversation. We relate the hand motion oscillatory gestures and symmetric gestures to the phases of speech and multimodal discourse analysis. We demonstrate the ability of our algorithm to extract gestural symmetries and oscillations and show how symmetric gestures and oscillatory gestures correspond to natural discourse structure.

[1]  Steven M. Seitz,et al.  View-Invariant Analysis of Cyclic Motion , 1997, International Journal of Computer Vision.

[2]  James W. Davis,et al.  Categorical representation and recognition of oscillatory motion patterns , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[3]  G. Revesz,et al.  [Thinking and speaking]. , 1950, Nederlands tijdschrift voor de psychologie en haar grensgebieden.

[4]  Ajit S. Bopardikar,et al.  Wavelet transforms - introduction to theory and applications , 1998 .

[5]  Randal C. Nelson,et al.  Detection and Recognition of Periodic, Nonrigid Motion , 1997, International Journal of Computer Vision.

[6]  William T. Freeman Computer vision for television and games , 1999, Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).

[7]  M. Studdert-Kennedy Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[8]  Daniel E. Koditschek,et al.  Dynamical system representation, generation, and recognition of basic oscillatory motion gestures , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[9]  Daniel E. Koditschek,et al.  Dynamic system representation of basic and nonlinear in parameters oscillatory motion gestures , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[10]  K. Goldstein,et al.  THINKING AND SPEAKING , 1960, Annals of the New York Academy of Sciences.

[11]  Julia Hirschberg,et al.  Instructions for annotating discourse , 1995 .

[12]  A. Grossmann,et al.  Cycle-octave and related transforms in seismic signal analysis , 1984 .

[13]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[14]  S. Mallat A wavelet tour of signal processing , 1998 .

[15]  Francis K. H. Quek,et al.  Gestural Origo and Loci-Transitions in Natural Discourse Segmentation , 2001 .

[16]  Fang Liu,et al.  Finding periodicity in space and time , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[17]  Francis Quek,et al.  A parallel algorithm for dynamic gesture tracking , 1999, Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).