On the Classification of Visual Languages by Grammar Hierarchies

Abstract Visual language specification has been investigated for more than two decades now and many different formalisms for specifying and parsing visual languages have been invented. However, there has been little attempt to develop a systematic and comprehensive hierarchy of visual languages based on their formal properties. Given the importance of the Chomsky hierarchy for the theory of textual languages and the difficulty of comparing these many different visual language formalisms, it is clear that there is a need for such a hierarchy. We develop a hierarchy for visual languages and investigate the expressiveness and cost of parsing for classes defined therein. Although the hierarchy is based on the constraint multiset grammar formalism, we sketch how other visual language specification formalisms can be mapped into constraint multiset grammars so that a comparison is made possible. One consequence of our work is that a large class of ‘naturally occurring’ visual languages are inherently context-sensitive, so that the core of such a hierarchy has to be built around different forms of context-sensitivity.