Automatic speech grammar generation during conceptual modelling of virtual environments

Speech interfaces are becoming more and more popular as a means to interact with virtual environments but the development and integration of these interfaces is usually still ad hoc, especially the speech grammar creation of the speech interface is a process commonly performed by hand. In this paper, we introduce an approach to automatically generate a speech grammar which is generated using semantic information. The semantic information is represented through ontologies and gathered from the conceptual modelling phase of the virtual environment application. The utterances of the user will be resolved using queries onto these ontologies such that the meaning of the utterance can be resolved. For validation purposes we augmented a city park designer with our approach. Informal tests validate our approach, because they reveal that users mainly use words represented in the semantic data, and therefore also words which are incorporated in the automatically generated speech grammar.

[1]  Vladimir Pavlovic,et al.  Speech/Gesture Interface to a Visual-Computing Environment , 2000, IEEE Computer Graphics and Applications.

[2]  Marc Erich Latoschik,et al.  Resolving object references in multimodal dialogues for immersive virtual environments , 2004, IEEE Virtual Reality 2004.

[3]  Philip R. Cohen,et al.  On the Relationships Among Speech, Gestures, and Object Manipulation in Virtual Environments: Initial Evidence , 2005 .

[4]  Karin Coninx,et al.  VR-DeMo: a Tool-supported Approach Facilitating Flexible Development of Virtual Environments using Conceptual Modelling , 2006 .

[5]  Scott McGlashan,et al.  Speech Interfaces to Virtual Reality , 1995 .

[6]  Karl-Hans Englmeier,et al.  Speech interaction in virtual reality , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[7]  Heedong Ko,et al.  Spatial ontology for semantic integration in 3D multimodal interaction framework , 2006, VRCIA '06.

[8]  Heedong Ko,et al.  Semantic 3D object manipulation using object ontology in multimodal interaction framework , 2005, ICAT '05.

[9]  Giuseppe Conti,et al.  "Verba Volant Scripta Manent" a false axiom within virtual environments. A semi-automatic tool for retrieval of semantics understanding for speech-enabled VR applications , 2006, Comput. Graph..

[10]  Karsten A. Otto.  The Semantics of Multi-user Virtual Environments , 2005 .

[11]  Minh Tue Vo,et al.  Building an application framework for speech and pen input integration in multimodal learning interfaces , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[12]  Steven K. Feiner,et al.  Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality , 2003, ICMI '03.

[13]  Ralph Grishman.  Proceedings of the fifth conference on Applied natural language processing , 1997 .

[14]  Liang Chen,et al.  QuickSet: Multimodal Interaction for Simulation Set-up and Control , 1997, ANLP.

[15]  K. Coninx,et al.  CoGenIVE : Code Generation for Interactive Virtual Environments , 2005 .

[16]  Barbara Hayes-Roth,et al.  An intelligent guide for virtual environments , 1997, AGENTS '97.

[17]  Deb Roy,et al.  Probabilistic grounding of situated speech using plan recognition and reference resolution , 2005, ICMI '05.

[18]  Philip R. Cohen.  The role of natural language in a multimodal interface , 1992, UIST '92.

[19]  Verba Volant, Scripta Manent , 2007 .