Towards a UML profile on formal semantics for modeling multimodal interactive systems - concepts for modeling interactive systems using standard tools of software engineering

An inherent general problem is that systems become more complex. This problem is exacerbated by the use of additional interaction concepts like multimodality. Interactive systems are widespread and often apply advanced interaction concepts to ease use and enhance user experience. Touch interaction and multimodality are concepts that are on the rise and are already offered by many commercial products. For example, user interfaces of the infotainment equipment in current premium cars provide speech interaction, especially to increase operational safety. A common state-of-the-art approach to master complexity comprising all phases of software development is Model-Driven Development. Modeling raises the level of abstraction by using (mainly graphical) models to bridge the gap between specification and implementation. The Unified Modeling Language (UML) is the de facto standard modeling language. Although UML has proven itself in practice, it does not support modeling of interactive systems and their user interfaces so far. We present an approach to extend UML for modeling interactive systems. We emphasize on modeling multimodality by using and extending UML state diagrams, by creating UML compliant extensions based on UML profiles, and by defining formal semantics for our extensions including behavioral aspects by means of Abstract State Machines. We propose an architecture using separate state diagrams for each modality. These state diagrams are synchronized by a common system model. The suitability, usability, and applicability of our approach is reviewed by means of an expert evaluation. Our approach enables modeling multimodal interactive systems with one formalism. Thus, it supports an integrated kind of modeling as well as separation of different concerns of multimodality. The provided formal semantics for our UML profile enables automated processing of our models including comprehensive tool support for simulation and code generation.