Automatic Hand Hold Detection in Natural Conversation

We present a motion-energy-based method of detecting hand holds in videos of natural conversations. The holds are found by classifying hand motions extracted from videos using computer vision techniques. We describe a set of heuristics for judging when a hold is detected and present empirical analysis of the efficacy of our algorithm against real video data that has been hand-coded for pauses. The quality of our detector is evaluated and its parameters optimized using a ROC Graph-based approach. The optimal hold detection heuristics are shown to closely reflect the observed physical properties of gestures.

[1]  J A Swets,et al.  Measuring the accuracy of diagnostic systems. , 1988, Science.

[2]  Scott K. Liddell,et al.  American Sign Language: The Phonological Base , 2013 .

[3]  Francis K. H. Quek,et al.  Toward a vision-based hand gesture interface , 1994 .

[4]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[5]  Sotaro Kita,et al.  Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders , 1997, Gesture Workshop.

[6]  Francis K. H. Quek,et al.  Vector Coherence Mapping: A Parallelizable Approach to Image Flow Computation , 1998, ACCV.

[7]  Dimitri Metaxas.  Deformable model and HMM-based tracking, analysis and recognition of gestures and faces , 1999, Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).

[8]  Ipke Wachsmuth,et al.  Coverbal iconic gestures for object descriptions in virtual environments , 1999 .

[9]  Francis Quek,et al.  A parallel algorithm for dynamic gesture tracking , 1999, Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).

[10]  Francis Quek,et al.  Gesture cues for conversational interaction in monocular video , 1999, Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).

[11]  Francis K. H. Quek,et al.  Gesture, speech, and gaze cues for discourse segmentation , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[12]  Francis K. H. Quek,et al.  Catchments, prosody and discourse , 2001 .