Multimodal Cooperation with the DENK System

In this chapter we present the DenK project, a long-term effort where the aim is to build a generic cooperative human-computer interface combining multiple input and output modalities. We discuss the view on human-computer interaction that underlies the project and the emerging DenK system. The project integrates results from fundamental research in knowledge representation, communication, natural language semantics and pragmatics, and object-oriented animation. Central stage in the project is occupied by the design of a cooperative and knowledge-able electronic assistant that communicates in natural language and that has internal access to an application domain which is presented visually to the user. The assistant, that we call the Cooperative Assistant, has an information state that is represented in a rich form of type theory, a formalism that enables us to model the inherent cognitive dynamics of a dialogue participant. This formalism is used both for modeling domain knowledge, for representing the current dialogue context, and for implementing a context-change theory of communication.

[1]  de Ng Dick Bruijn,et al.  A survey of the project Automath , 1980 .

[2]  Harry Bunt,et al.  Context and Dialogue Control , 1994 .

[3]  Bob Carpenter,et al.  The logic of typed feature structures , 1992 .

[4]  Vaj Tijn Borghuis.  Coming to terms with modal logic : on the interpretation of modalities in typed lambda-calculus , 1994 .

[5]  ScopingsJerry R. Hobbs,et al.  An Algorithm for Generating Quanti er , 1987 .

[6]  William C. Frederick,et al.  A Combinatory Logic , 1995 .

[7]  Clarisse Sieckenius de Souza,et al.  The Semiotic Engineering of User Interface Languages , 1993, Int. J. Man Mach. Stud..

[8]  H. C. Bunt,et al.  DIT: Dynamic interpretation in text and dialogue , 1990 .

[9]  L. C. van der Gaag,et al.  Monotonic descriptions of Dynamic Worlds , 1996 .

[10]  René M. C. Ahn.  Communication Contexts: a Pragmatic Approach to Information Exchange , 1994, TYPES.

[11]  Wolfgang Wahlster,et al.  Plan-Based Integration of Natural Language and Graphics Generation , 1993, Artif. Intell..

[12]  Hendrik Pieter Barendregt,et al.  Introduction to generalized type systems , 1991, Journal of Functional Programming.

[13]  Tijn Borghuis,et al.  Interpreting Modal Natural Deduction in Type Theory , 1993 .

[14]  Robbert-Jan Beun,et al.  The recognition of declarative questions in information dialogues , 1989 .

[15]  Eaj Eric Peeters.  Design of an object-oriented, interactive animation system , 1995 .

[16]  John D. Burger,et al.  The Application of Natural Language Models to Intelligent Multimedia , 1991, AAAI Workshop on Intelligent Multimedia Interfaces.

[17]  Stuart C. Shapiro,et al.  Intelligent multi-media interface technology , 1991 .

[18]  J. Allwood.  Linguistic communication as action and cooperation : a study in pragmatics , 1976 .

[19]  R. J. Beun.  A framework for cooperative dialogues , 1991 .

[20]  Ivan A. Sag,et al.  Book Reviews: Head-driven Phrase Structure Grammar and German in Head-driven Phrase-structure Grammar , 1996, CL.

[21]  Uwe Reyle,et al.  From Discourse to Logic - Introduction to Modeltheoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory , 1993, Studies in linguistics and philosophy.

[22]  Johannes Ogier Maria Jaspars,et al.  Calculi for constructive communication : a study of the dynamics of partial states , 1994 .

[23]  Leen Helmink,et al.  Goal directed proof construction in type theory , 1991 .

[24]  M.E.M.C. Verlinden.  Development of the core syntax for DenK , 1996 .

[25]  Per Martin-Löf,et al.  Intuitionistic type theory , 1984, Studies in proof theory.

[26]  Pierre Falzon,et al.  AN ARCHITECTURE FOR MULTIMODAL DIALOGUE , 1991 .

[27]  F. Dick.  A survey of the project Automath , 1980 .

[28]  H. Alshawi,et al.  The Core Language Engine , 1994 .

[29]  Anita H. M. Cremers.  Object Reference in Task-Oriented Keyboard Dialogues , 1995, Multimodal Human-Computer Communication.

[30]  H. C. Bunt.  Dialogue control functions and interaction design , 1994 .

[31]  Cornelius W. A. M. van Overveld.  The generalized display processors as an approach to real-time interactive 3-D computer animation , 1991, Comput. Animat. Virtual Worlds.

[32]  Harry Bunt,et al.  Dialogue Context modeling , 1997 .

[33]  Edwin Hutchins,et al.  Metaphors for Interface Design , 1987 .

[34]  R. Ahn,et al.  Discourse representation meets constructive mathematics , 1990 .

[35]  Jerry R. Hobbs,et al.  An Algorithm for Generating Quantifier Scopings , 1987, CL.