Knowledge Bases and Tools for Building Integrated Multimedia Intelligent Interfaces

A multimedia interface is one that supports more than one medium through which users and application programs can communicate. For example, such an interface might support graphics, gestures, natural language, menus, sketching, and video. An intelligent interface is one that exploits knowledge about the task, the application, the interface, and the user in ways that help the user accomplish the task effectively. This includes interpreting ambiguous inputs correctly in the current context, phrasing outputs appropriately for the current user, and providing advice on efficient ways to use the system to accomplish the user's goals. An integrated interface is one in which events and objects in one part of the interface are accessible in the other parts so that a single task can be split across interface components as appropriate. In this paper, we present an architecture for an integrated set of tools now under development, called Human Interface Tools, or HITS. These tools are intended to support both the construction and the run-time execution of integrated multimedia intelligent interfaces. This paper describes our design of HITS, with some examples from parts of the system that have already been implemented. We will not attempt to articulate clearly the line between the current working toolset and our vision of a more complete system but will summarize the current state of the system at the close of the paper.