Automatic extraction of semantic information for a context sensitive multimodal framework for VR

The capability of processing spoken commands is one of the most important features of modern multimodal AR/VR environments. This feature requires programmers to compile some human supplied knowledge in the form of grammars which are used at runtime to process spoken utterances into complete commands. Further speech recognition (SR) must be hard-coded into the application. This time-consuming, error-prone process is repeated every time modifications to the code are introduced. This paper presents a completely automatic process to build a body of knowledge from the information embedded within the application source code. The programmer in fact often embeds, throughout the coding process, a vast amount of semantic information by defining classes, reference names, or through method definitions. This research work exploits this semantic richness and it provides a self-configurable system, which automatically adapts its understanding of human commands according to the semantic information within the application's source code

[1]  Alberto J. Cañas,et al.  Using WordNet for Word Sense Disambiguation to Support Concept Map Construction , 2003, SPIRE.

[2]  Hannes Kaufmann,et al.  Collaborative Augmented Reality in Education , 2002 .

[3]  George A. Miller,et al.  Introduction to WordNet: An On-line Lexical Database , 1990 .

[4]  Scott McGlashan,et al.  Speech Interfaces to Virtual Reality , 1995 .

[5]  Mark Billinghurst,et al.  The use of dense stereo range data in augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[6]  R. Darnell.  Translation , 1873, The Indian medical gazette.

[7]  Thomas Hofmann,et al.  Hierarchical Semantic Classification: Word Sense Disambiguation with World Knowledge , 2003, IJCAI.

[8]  George A. Miller,et al.  Nouns in WordNet: A Lexical Inheritance System , 1990 .

[9]  Bernd Schwald,et al.  An Augmented Reality System for Training and Assistence to Maintenance in the Industrial Context , 2003, WSCG.

[10]  Giuseppe Conti,et al.  Tangible interfaces in virtual environments for industrial design , 2004, AVI.

[11]  Sharon L. Oviatt,et al.  Multimodal Interaction for 2D and 3D Environments , 1999, IEEE Computer Graphics and Applications.

[12]  Steven K. Feiner,et al.  Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality , 2003, ICMI '03.

[13]  Nancy Ide,et al.  Introduction to the Special Issue on Word Sense Disambiguation: The State of the Art , 1998, Comput. Linguistics.

[14]  Christiane Fellbaum,et al.  English Verbs as a Semantic Net , 1990 .

[15]  Dieter Schmalstieg,et al.  OpenTracker-an open software architecture for reconfigurable tracking based on XML , 2001, Proceedings IEEE Virtual Reality 2001.

[16]  Dieter Schmalstieg,et al.  “Studierstube”: An environment for collaboration in augmented reality , 1998, Virtual Reality.

[17]  Christian Sandor,et al.  Towards a Development Methodology for Augmented Reality User Interfaces , 2004, MIXER.

[18]  Marc Erich Latoschik.  A gesture processing framework for multimodal interaction in virtual reality , 2001, AFRIGRAPH '01.

[19]  Sharon L. Oviatt,et al.  Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions , 2000, Hum. Comput. Interact..

[20]  Joseph J. LaViola.  Msvt: a virtual reality-based multimodal scientific visualization tool , 1999 .

[21]  Michael Sussna,et al.  Word sense disambiguation for free-text indexing using a massive semantic network , 1993, CIKM '93.

[22]  Rada Mihalcea,et al.  A Method for Word Sense Disambiguation of Unrestricted Text , 1999, ACL.

[23]  Didier Stricker,et al.  Augmented reality for construction tasks: doorlock assembly , 1999 .

[24]  Augmented Reality in the Palm of your Hand : A PDA-Based Framework Offering a Location-based , 3 D and Speech-Driven User Interface , 2003 .

[25]  Stan Matwin,et al.  A WordNet-based Algorithm for Word Sense Disambiguation , 1995, IJCAI.

[26]  Hang Li,et al.  Word Translation Disambiguation Using Bilingual Bootstrapping , 2002, ACL.