Towards a Unified Knowledge-Based Approach to Modality Choice

This paper advances a unified knowledge-based approach to the process of choosing the most appropriate modality or combination of modalities in multimodal output generation. We propose a Modality Ontology (MO) that models the knowledge needed to support the two most fundamental processes determining modality choice – modality allocation (choosing the modality or set of modalities that can best support a particular type of information) and modality combination (selecting an optimal final combination of modalities). In the proposed ontology we model the main levels which collectively determine the characteristics of each modality and the specific relationships between different modalities that are important for multi-modal meaning making. This ontology aims to support the automatic selection of modalities and combinations of modalities that are suitable to convey the meaning of the intended message.

[1]  Niels Ole Bernsen,et al.  Foundations of Multimodal Representations: A Taxonomy of Representational Modalities , 1994, Interact. Comput..

[2]  Jane Hunter,et al.  Adding Multimedia to the SemanticWeb: Building and Applying an MPEG-7 Ontology , 2005, Multimedia Content and the Semantic Web.

[3]  P. Chandler,et al.  Cognitive Load Theory and the Format of Instruction , 1991 .

[4]  Edward Rolf Tufte,et al.  The visual display of quantitative information , 1985 .

[5]  Barbara Tversky,et al.  Animation: can it facilitate? , 2002, Int. J. Hum. Comput. Stud..

[6]  Hartmut Stöckl,et al.  In between modes: Language and image in printed media , 2004 .

[7]  Jon Oberlander,et al.  Reasoning with Words, Pictures and Calculi: Computation Versus Justification , 1991 .

[8]  P. Fayers,et al.  The Visual Display of Quantitative Information , 1990 .

[9]  Stefanos Kollias,et al.  Multimedia Content and the Semantic Web: Standards, Methods and Tools , 2005 .

[10]  Michael Twyman,et al.  A Schema for the Study of Graphic Language (Tutorial Paper) , 1979 .

[11]  R. Mayer,et al.  Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity , 1999 .

[12]  Edward R. Tufte,et al.  The Visual Display of Quantitative Information , 1986 .

[13]  Neff Walker,et al.  A classification of visual representations , 1994, CACM.

[14]  M. D’Esposito Working memory. , 2008, Handbook of clinical neurology.

[15]  Eduard H. Hovy,et al.  On the Knowledge Underlying Multimedia Presentations , 1991, AAAI Workshop on Intelligent Multimedia Interfaces.