A syntax directed approach to picture semantics

If one constructs a visual language, it is necessary to have a methodology for constructing the relationship between aspects of the pictures and the corresponding aspects of the application domain. The various methods for specifying Visual Languages emphasize computational aspects, where the semantics of the picture is embedded in the computational setting of the specification, but they are not based on an understanding of the cognitive issues involved in the semantics of pictures and in the use of the pictures for a better grasp of the application domain and for manipulating the domain. We think that the analogy between a picture and its meaning is what can help people to understand the meaning represented by the picture, and that the match between the syntactic structures of the picture and what it represents is an important element of analogy. We formalise the notion of matching in an approach to picture semantics based on order-sorted algebra. Pictures are described in a well-structured framework (order sorted signature) and so is the application domain. Constructing the relationship between pictures and their meanings is guided by a formal notion of signature morphism which, combining with the formal description of pictures, enforces a structural match between pictures and their represented. We also discuss the various issues brought out by this algebra approach.