A Model for Multimodal Representation and Inference

In this paper some a pplications of a theory for r epresentation and inference in multimodal scenarios is presented. The theory is focused on the relation between natural language a nd graphical expressions. A basic assumption is that graphical expressions belong to a language with well-defined syntax and semantics: a graphical l anguage. A second assumption is that the relation between expressions of different modalities is similar to the relation of translation that holds between expressions of different natural languages. In this paper a multimodal system of representation and inference based o n this view of modality is described. First, a brief introduction to the representational structures of the multimodal system is presented. Then, a number of multimodal inferences s upported b y the system are illustrated. These e xamples show how the multimodal system of representation can support t he definition and u se of graphical languages, perceptual inferences for problem-solving and interpretation of multimodal messages. Finally, the intuitive notion of modality underlying this research is discussed. 1. Multimodal Representation The system of multimodal representation that is s ummarized in this paper is illustrated in Figure 1. The notion of modality in which the system is based is a representational notion: information conveyed in one particular modality is expressed in a representational l anguage a ssociated with the modality. Each modality in the system is captured through a particular language, and relations between expressions of different modalities are captured in terms of translation functions from basic and composite expressions of the source modality into expressions of the object modality. This view of multimodal representation and reasoning h as been developed in [13], [17], [9], [18] and [19], and it follows closely the spirit of Montague’s general semiotic programme [5].