Natural multimodal interaction for design applications

One of the open issues in multimodal interaction is the antagonism between proponents of the direct manipulation and the communication agent metaphors. In this paper we describe an experiment in which we tested a system based on the communication agent metaphor for a bathroom design application where direct manipulation is possible in principle, but perhaps not very attractive in terms of usability. We found that non-expert users have substantial difficulties interacting with the direct manipulation systems, due to the fact that they failed to discover the mental model applied by the application designers. This problem did not occur in the communication agent application, but here we observed the need for substantial improvements of the recognition technology needed to build conversational interfaces.