Modeling and Processing of Oral and Tactile Activities in the GEORAL System

This paper presents the processing of tactile and oral multimodal events that occur in inputs of the GEORAL tactile system. First we shall describe the principal results of observations on the behaviour of users in a multimodal interaction environment. These observations, combined with consideration of the technical possibilities and limitations of speech recognition, lead us to proposing a system architecture. The chapter ends with details of the modeling of communicative acts, based on plan operators, and the principal stages of the merging of tactile and oral acts.

[1]  Sharon L. Oviatt,et al.  A rapid semi-automatic simulation technique for investigating interactive speech and handwriting , 1992, ICSLP.

[2]  Philip R. Cohen.  The role of natural language in a multimodal interface , 1992, UIST '92.

[3]  Steven K. Feiner,et al.  Coordinating Text and Graphics in Explanation Generation , 1989, HLT.

[4]  Mark T. Maybury,et al.  Planning Multimedia Explanations Using Communicative Acts , 1991, AAAI Workshop on Intelligent Multimedia Interfaces.

[5]  James F. Allen,et al.  A Plan Recognition Model for Subdialogues in Conversations , 1987, Cogn. Sci..

[6]  D. Bouwhuis,et al.  The Structure of Multimodal Dialogue , 1989 .

[7]  Georgia M. Green.  Pragmatics and Natural Language Understanding , 1989 .

[8]  Carlo Strapparava,et al.  Dialogue Cohesion Sharing and Adjusting in an Enhanced Multimodal Environment , 1993, IJCAI.

[9]  Marie-Luce Bourguet.  Conception et réalisation d'une interface de dialogue personne-machine multimodale , 1992 .

[10]  Mark T. Maybury,et al.  Communicative Acts for Explanation Generation , 1992, Int. J. Man Mach. Stud..

[11]  James F. Allen,et al.  A Plan Recognition Model for Subdialogues in Conversations , 1987, Cogn. Sci..

[12]  Philip R. Cohen.  The Pragmatics of Referring and the Modality of Communication , 1984, Comput. Linguistics.

[13]  James F. Allen.  Natural language understanding , 1987, Bejnamin/Cummings series in computer science.

[14]  Wolfgang Wahlster,et al.  Designing Illustrated Texts: How Language Production Is Influenced by Graphics Generation , 1991, EACL.