Multi-Modal-Method: A Design Method for Building Multi-Modal Systems

This paper describes Multi-Modal-Method, a design method for building grammar-based multi modal systems. Multi-Modal-Method defines the procedure which interface designers may follow in developing multi-modal systems, and provides MM-DCG, a grammatical framework for multi-modal input interpretation. Multi-Modal-Method has been inductively defined through several experimental multi-modal interface system developments. A case study of a multi-modal drawing tool development along with Multi-Modal-Method is reported.

[2]  Hideo Shimazu,et al.  Drawing Pictures with Natural Language and Direct Manipulation , 1994, COLING.

[3]  Norbert Reithinger,et al.  Bidirectional Use of Knowledge in the Multi-modal NL Access System XTRA , 1989, IJCAI.

[4]  Joëlle Coutaz,et al.  A design space for multimodal systems: concurrent processing and data fusion , 1993, INTERCHI.

[5]  Yacine Bellik,et al.  A multimodal dialogue controller for multimodal user interface management system application: a multimodal window manager , 1993, CHI '93.

[6]  Alexander H. Waibel,et al.  Multi-modal HCI: combination of gesture and speech recognition , 1993, CHI '93.

[7]  Philip R. Cohen.  The role of natural language in a multimodal interface , 1992, UIST '92.

[8]  Seigo Arita,et al.  Multi-Modal Definite Clause Grammar , 1994, COLING 1994.

[9]  Philip R. Cohen,et al.  Synergistic use of direct manipulation and natural language , 1989, CHI '89.

[10]  Philip J. Hayes,et al.  Steps towards Integrating Natural Language and Graphical Interaction for Knowledge-Based Systems , 1986, European Conference on Artificial Intelligence.

[11]  David H. D. Warren,et al.  Definite Clause Grammars for Language Analysis - A Survey of the Formalism and a Comparison with Augmented Transition Networks , 1980, Artif. Intell..

[12]  Wolfgang Wahlster,et al.  User and discourse models for multimodal communication , 1991 .

[13]  Wolfgang Wahlster,et al.  Combining Deictic Gestures and Natural Language for Referent Identification , 1986, COLING.