Formal description techniques to support the design, construction and evaluation of fusion engines for sure (safe, usable, reliable and evolvable) multimodal interfaces

Representing the behaviour of multimodal interactive systems in a complete, concise and non-ambiguous way is still a challenge for formal description techniques (FDT). Depending on the FDT, multimodal interactive systems feature specific characteristics that are either cumbersome or impossible to capture with classical FDT. This is due to the multiple (potentially synergistic) use of modalities and the strong temporal constraints usually encountered in this kind of systems that have to be dealt with exhaustively if FDT are used. This paper focuses on the requirements for the modelling and construction of fusion engines for multimodal interfaces. It proposes a formal description technique dedicated to the engineering of interactive multimodal systems able to address the challenges of fusion engines. Such benefits are presented on a set of examples illustrating both the constructs and the process.

[1]  Philippe A. Palanque,et al.  A model-based tool for interactive prototyping of highly interactive applications , 2002, CHI Extended Abstracts.

[2]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[3]  Philippe A. Palanque,et al.  Petri net objects for the design, validation and prototyping of user-driven interfaces , 1990, INTERACT.

[4]  Philippe A. Palanque,et al.  Fusion engines for multimodal input: a survey , 2009, ICMI-MLMI '09.

[5]  Marc Erich Latoschik.  Designing transition networks for multimodal VR-interactions using a markup language , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[6]  Marco Winckler,et al.  Tailoring Usability into Agile Software Development Projects , 2008, Maturing Usability.

[7]  Ann Blandford,et al.  Four easy pieces for assessing the usability of multimodal interaction: the CARE properties , 1995, INTERACT.

[8]  David Harel,et al.  Statecharts: A Visual Formalism for Complex Systems , 1987, Sci. Comput. Program..

[9]  Mohammed Yeasin,et al.  A real-time framework for natural multimodal interaction with large screen displays , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[10]  Eric Barboni,et al.  Model-Based Engineering of Widgets, User Applications and Servers Compliant with ARINC 661 Specification , 2006, DSV-IS.

[11]  Thierry Ganille,et al.  ICARE software components for rapidly developing multimodal interfaces , 2004, ICMI '04.

[12]  Robert J. K. Jacob,et al.  A software model and specification language for non-WIMP user interfaces , 1999, TCHI.

[13]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[14]  Mike Sinclair,et al.  Interaction and modeling techniques for desktop two-handed input , 1998, UIST '98.

[15]  Asaf Degani,et al.  Formal Analysis and Automatic Generation of User Interfaces: Approach, Methodology, and an Algorithm , 2007, Hum. Factors.

[16]  Michael Johnston,et al.  Finite-state multimodal integration and understanding , 2005, Natural Language Engineering.

[17]  Stephen J. Mellor,et al.  Executable UML - A Foundation for Model-Driven Architecture , 2002, Addison Wesley object technology series.

[18]  Ted G. Lewis,et al.  Direct-manipulation user interface modeling with high-level Petri nets , 1991, CSC '91.

[19]  Philippe A. Palanque,et al.  Design, specification, and verification of interactive systems , 2001, SOEN.

[20]  Philip R. Cohen,et al.  QuickSet: multimodal interaction for distributed applications , 1997, MULTIMEDIA '97.

[21]  Fang Chen,et al.  A novel method for multi-sensory data fusion in multimodal human computer interaction , 2006, OZCHI.

[22]  Joëlle Coutaz,et al.  A generic platform for addressing the multimodal challenge , 1995, CHI '95.

[23]  David L. Parnas,et al.  On the use of transition diagrams in the design of a user interface for an interactive computer system , 1969, ACM '69.

[24]  Mark Green,et al.  A survey of three dialogue models , 1986, TOGS.

[25]  Philippe A. Palanque,et al.  A model-based approach for real-time embedded multimodal systems in military aircrafts , 2004, ICMI '04.

[26]  C. Y. Thielman,et al.  Natural Language with Integrated Deictic and Graphic Gestures , 1989, HLT.

[27]  Marco Winckler,et al.  Usability Study of Multi-modal Interfaces Using Eye-Tracking , 2007, INTERACT.

[28]  Marco Winckler,et al.  StateWebCharts: A Formal Description Technique Dedicated to Navigation Modelling of Web Applications , 2003, DSV-IS.

[29]  Michael D. Harrison,et al.  Prototyping Pre-implementation Designs of Virtual Environment Behaviour , 2001, EHCI.

[30]  Marco Winckler,et al.  A Formal Description of Multimodal Interaction Techniques for Immersive Virtual Reality Applications , 2005, INTERACT.

[31]  Eric Barboni,et al.  High-Fidelity Prototyping of Interactive Systems Can Be Formal Too , 2009, HCI.

[32]  Ivan Marsic,et al.  A framework for rapid development of multimodal interfaces , 2003, ICMI '03.