Temporal Symbolic Integration Applied to a Multimodal System Using Gestures and Speech

This paper presents a technical approach for temporal symbol integration aimed to be generally applicable in unimodal and multimodal user interfaces. It draws its strength from symbolic data representation and an underlying rule-based system, and is embedded in a multiagent system. The core method for temporal integration is motivated by findings from cognitive science research. We discuss its application for a gesture recognition task and speech-gesture integration in a Virtual Construction scenario. Finally an outlook of an empirical evaluation is given.

[1]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[2]  Keisoku Jido Seigyo Gakkai Staff.  IECON '98: Proceedings of the 24th Annual Conference of the IEEE Industrial Electronics Society Aachen, Germany, August 31-September 4, 1998 / , 1998 .

[3]  E. Pöppel,et al.  A hierarchical model of temporal perception , 1997, Trends in Cognitive Sciences.

[4]  Ipke Wachsmuth,et al.  Gesture and Sign Language in Human-Computer Interaction , 1997, Lecture Notes in Computer Science.

[5]  Sharon L. Oviatt,et al.  Unification-based Multimodal Integration , 1997, ACL.

[6]  Alistair D. N. Edwards,et al.  Hand Tension as a Gesture Segmentation Cue , 1996, Gesture Workshop.

[7]  Alan Wexelblat.  Research Challenges in Gesture: Open Issues and Unsolved Problems , 1997, Gesture Workshop.

[8]  Ipke Wachsmuth,et al.  Communicative Rhythm in Gesture and Speech , 1999, Gesture Workshop.

[9]  Britta Lenzmann,et al.  Benutzeradaptive und multimodale Interface-Agenten , 1998, DISKI.

[10]  C. Collin,et al.  An Introduction to Natural Computation , 1998, Trends in Cognitive Sciences.

[11]  Marc Erich Latoschik,et al.  Utilize speech and gestures to realize natural interaction in a virtual environment , 1998, IECON '98. Proceedings of the 24th Annual Conference of the IEEE Industrial Electronics Society (Cat. No.98CH36200).

[12]  Marc Erich Latoschik,et al.  Exploiting Distant Pointing Gestures for Object Selection in a Virtual Environment , 1997, Gesture Workshop.

[13]  Ipke Wachsmuth,et al.  Gesture Recognition of the Upper Limbs - From Signal to Symbol , 1997, Gesture Workshop.

[14]  Joëlle Coutaz,et al.  A generic platform for addressing the multimodal challenge , 1995, CHI '95.

[15]  Marc Erich Latoschik,et al.  Knowledge-based assembly simulation for virtual prototype modeling , 1998, IECON '98. Proceedings of the 24th Annual Conference of the IEEE Industrial Electronics Society (Cat. No.98CH36200).

[16]  Franz Kummert,et al.  Incremental speech recognition for multimodal interfaces , 1998, IECON '98. Proceedings of the 24th Annual Conference of the IEEE Industrial Electronics Society (Cat. No.98CH36200).