A Framework for Constructing Visualization, Animation, and Direct Manipulation Interfaces

This thesis describes a framework for developing kinds of non-WIMP-based user interface software, designated as a bi-directional translation model1. Here, ‘non-WIMP-based’ interface means that it cannot built only by combining WIMP widgets, such as buttons and menus. In particular, the targets of this thesis are two types of GUI software. One is direct manipulation interfaces for figures and diagrams that visually represent various abstract objects and relations in an application data, such as direct manipulation interfaces for network and hierarchical data structures. They enable the user to draw and modify diagrams to input abstract data into an application. Another is animations of changing abstract objects and relations in an application data, such as animations of sorting algorithms. They show their transitions visually, and are useful for understanding the behavior of an application or an algorithm. The bi-directional translation model models the general process of visualization, picture recognition, and animation generation. It is originated in Kamada’s general framework for visualizing abstract objects and relations. We have extended it by integrating recognition of figures/diagrams and animation of changing abstract objects and relations, which is one of the contributions in this thesis. The key idea of the framework is that these functions are translations between different data representations. The model introduced four data representations: AR (Application Data Representation), ASR (Abstract Structure Representation), VSR (Visual Structure Representation), and PR (Pictorial Representation). In the model, visualization is a translation from AR to PR via ASR and VSR. Recognition of figures is an inverse translation from PR to AR via VSR and ASR. The essential translation is that between ASR and VSR, because there is no a priori mapping between these representations. ASR is the representation in our model that represents the structure of application data, and it does not have explicit visual appearance information. On the other hand, the purpose of VSR is to represent the high-level structures of pictures. The mapping between ASR and VSR is the essential conversion between abstract data for an application and pictorial data for presentation. Nevertheless, the structure of ASR and that of VSR are usually similar. As users understand the meaning of abstract application data via pictures, visualized pictures should represent abstract application data, and thus they should have a similar structure. The programmer specifies these translations by visual mapping rule sets, which are declarative rules that define mappings between ASR data and VSR data. As both structures are similar, mapping rules are usually simple. Animations are also handled in the framework extended naturally to the time dimension. In the framework, animations are regarded as operations on PR that are translated from operations on AR via operations on ASR and VSR. The translation among operations is executed maintaining consistency with the mapping relations among the AR, ASR, VSR, and PR data. We have chosen an interpolation-based method for implementation of the extended framework for animations. That is, animations are generated by interpolating a sequence of pictures translated from the running application’s internal data. Rather than specifying animations (motions or transformations) directly, the WIMP stands for Windows, Icons, Menus and Pointing device.