A Model-based Approach for Engineering Multimodal Mobile Interactions

Mobile phones have benefited from many enhancements since several years, including an increasing number of their embedded sensors (accelerometer, light sensor...). Created for giving context-aware abilities, these sensors also allow new types of interaction and pave the way to a great expansion of multimodal mobile interactions. Unfortunately, the current context of mobile software development makes difficult the development of multimodal applications. We propose our model-based solution that aims to facilitate the development of multimodal mobile interfaces. We adopt the principles of the Model-Driven Engineering (MDE) which is particularly fitted for such context. Our proposition includes M4L modeling language, library of high-level sensor-based modalities and MIMIC framework. Our approach respects the main criteria of the MDE in order to define an efficient model-based approach. Results of the first evaluation suggest that using our approach facilitates the development of sensor-based multimodal mobile interfaces.

[1]  Tarun Jaiswal,et al.  Fundamentals of Software Engineering , 2017, Lecture Notes in Computer Science.

[2]  Karin Coninx,et al.  Tool support for designing context-sensitive user interfaces using a model-based approach , 2005, TAMODIA '05.

[3]  Gregor Mehlmann,et al.  Modeling parallel state charts for multithreaded multimodal dialogues , 2011, ICMI '11.

[5]  Elisabeth André,et al.  Tool-Supported User-Centred Prototyping of Mobile Applications , 2011, Int. J. Handheld Comput. Res..

[6]  Fabio Paternò,et al.  Generation of Multi-Device Adaptive MultiModal Web Applications , 2013, MobiWIS.

[7]  Ulrich Frank,et al.  Some Guidelines for the Conception of Domain-Specific Modelling Languages , 2011, EMISA.

[8]  Douglas C. Schmidt,et al.  System Integration Using Model-Driven Engineering , 2009 .

[9]  Ann Blandford,et al.  Four easy pieces for assessing the usability of multimodal interaction: the CARE properties , 1995, INTERACT.

[10]  Anneke Kleppe,et al.  MDA explained - the Model Driven Architecture: practice and promise , 2003, Addison Wesley object technology series.

[11]  Marie-Luce Bourguet,et al.  A Toolkit for Creating and Testing Multimodal Interface Designs , 2002 .

[12]  Mark Rouncefield,et al.  Model-driven engineering practices in industry , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[13]  Marcos Serrano,et al.  Multimodal interaction on mobile phones: development and evaluation using ACICARE , 2006, Mobile HCI.

[14]  Stuart Kent,et al.  Model Driven Engineering , 2002, IFM.

[15]  Michael D. Harrison,et al.  Model Checking Interactor Specifications , 2001, Automated Software Engineering.

[16]  Bernhard Rumpe,et al.  Model-driven Development of Complex Software : A Research Roadmap , 2007 .

[17]  Jean-Claude Martin.  TYCOON: six primitive types of cooperation for observing, evaluating and specifying cooperations , 1999 .

[18]  Yacine Bellik,et al.  A framework for the intelligent multimodal presentation of information , 2006, Signal Process..

[19]  Jean-Yves Lionel Lawson,et al.  The openinterface framework: a tool for multimodal interaction. , 2008, CHI Extended Abstracts.

[20]  Daniel L. Moody,et al.  The “Physics” of Notations: Toward a Scientific Basis for Constructing Visual Notations in Software Engineering , 2009, IEEE Transactions on Software Engineering.

[21]  Denis Lalanne,et al.  Description languages for multimodal interaction: a set of guidelines and its illustration with SMUIML , 2010, Journal on Multimodal User Interfaces.

[22]  Roman Rädle,et al.  Squidy: a zoomable design environment for natural user interfaces , 2009, CHI Extended Abstracts.

[23]  Douglas C. Schmidt,et al.  Guest Editor's Introduction: Model-Driven Engineering , 2006, Computer.

[24]  Antonio Vallecillo,et al.  Formal and Tool Support for Model Driven Engineering with Maude , 2007, J. Object Technol..

[25]  Michel R. V. Chaudron,et al.  Managing Model Quality in UML-Based Software Development , 2005, 13th IEEE International Workshop on Software Technology and Engineering Practice (STEP'05).

[26]  Laurence Nigay,et al.  ICARE: a component-based approach for the design and development of multimodal interfaces , 2004, CHI EA '04.

[27]  Parastoo Mohagheghi.  Evaluating Software Development Methodologies Based on their Practices and Promises , 2008, SoMeT.

[28]  Ramanathan V. Guha,et al.  Cyc: toward programs with common sense , 1990, CACM.

[29]  Marc Erich Latoschik.  Designing transition networks for multimodal VR-interactions using a markup language , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[30]  Alessandro Valli,et al.  The design of natural interaction , 2008, Multimedia Tools and Applications.

[31]  Xavier Le Pallec,et al.  Multimodal interaction: a survey from model driven engineering and mobile perspectives , 2013, Journal on Multimodal User Interfaces.