Interpretation of Shape-Related Iconic Gestures in Virtual Environments

So far, approaches towards gesture recognition focused mainly on deictic and emblematic gestures. Iconics, viewed as iconic signs in the sense of Peirce, are different from deictics and emblems, for their relation to the referent is based on similarity. In the work reported here, the breakdown of the complex notion of similarity provides the key idea towards a computational model of gesture semantics for iconic gestures. Based on an empirical study, we describe first steps towards a recognition model for shape-related iconic gestures and its implementation in a prototype gesture recognition system. Observations are focused on spatial concepts and their relation to features of iconic gestural expressions. The recognition model is based on a graph-matching method which compares the decomposed geometrical structures of gesture and object.

[1]  Stefan Kopp,et al.  A Knowledge-based Approach for Lifelike Gesture Animation , 2000, ECAI.

[2]  B. Butterworth,et al.  Iconic gestures, imagery, and word retrieval in speech , 1997 .

[3]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[4]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Lambert Schomaker,et al.  Audio visual and Multimodal Speech Systems , 2003 .

[6]  Marc Erich Latoschik.  A General Framework for Multimodal Interaction in Virtual Reality Systems: PrOSA , 2001 .

[7]  Sotaro Kita,et al.  How representational gestures help speaking , 2000 .

[8]  Pierre Feyereisen,et al.  The Meaning of Gestures - What Can Be Understood Without Speech , 1988 .

[9]  C. Hartshorne,et al.  Collected Papers of Charles Sanders Peirce , 1935, Nature.

[10]  Ipke Wachsmuth,et al.  Coverbal iconic gestures for object descriptions in virtual environments , 1999 .

[11]  C.C.M. Hummels,et al.  Gestural design tools: Prototypes, experiments and scenarios , 2000 .

[12]  C. Peirce,et al.  Collected Papers of Charles Sanders Peirce , 1936, Nature.

[13]  Henrik Tramberend,et al.  Avocado: a distributed virtual reality framework , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[14]  Francis K. H. Quek,et al.  Gesture and Speech Multimodal Conversational Interaction , 2001 .

[15]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[16]  A. Lecours,et al.  The Biological foundations of gestures : motor and semiotic aspects , 1986 .

[17]  David McNeill,et al.  Language and Gesture: Catchments and contexts: non-modular factors in speech and gesture production , 2000 .