AN ARCHITECTURE FOR MULTIMODAL DIALOGUE

This paper describes the architecture of the MMI 2 system (Man-Machine Interface for MultiModal Interaction with knowledge based systems). The objective of the MMI 2 project is to develop a toolkit and a method for using the toolkit to construct knowledge based systems (KBS) in different domains which have multimodal interfaces and undertake cooperative dialogue with users. Within the project, a step towards the development of this toolkit is the development of a demonstrator KBS in the domain of computer network design. The project is structured so that the demonstration KBS will act as a shell which contains knowledge about the computer network design domain for the demonstration application, but which can also be used later for other domains when the requisite knowledge about those domains is encoded. The MMI2 project makes a strong commitment to “multimodal” rather than to “multimedia” interaction in the interface. The distinction intended is that a multi-media system is one which uses different presentation media (e.g. text, raster graphics, video, speech) without a commitment to the underlying representation of the information presented. For reasons of efficiency of both storage and processing, individual specialised representations are usually used for information which is intended to be presented in each individual medium, and information can only be presented in a single medium. A “multimodal” system is one which includes several input and output media, but is committed to a single internal representation language for the information. This permits the same information to be presented in any mode, chosen purely by rules which select that mode for that user at that point in task performance as being both sufficiently expressive and most efficient. The modes available in the MMI 2 demonstrator are: for input: English, French and Spanish natural languages, gesture, direct manipulation of graphics, command language; and for output: English, French and Spanish natural languages, graphics (CAD diagrams and business graphics), and non-verbal audio. The representation language used for all information within the system is a promiscuously reified typed logic called the Common Meaning Representation (CMR).