Formal description of multi-touch interactions

The widespread use of multi-touch devices and the large amount of research that has been carried out around them has made this technology mature in a very short amount of time. This makes it possible to consider multi-touch interactions in the context of safety critical systems. Indeed, beyond this technical aspect, multi-touch interactions present significant benefits such as input-output integration, reduction of physical space, sophisticated multi-modal interaction? However, interactive cockpits belonging to the class of safety critical systems, development processes and methods used in the mass market industry are not suitable as they usually focus on usability and user experience factors upstaging dependability. This paper presents a tool-supported model-based approach suitable for the development of interactive systems featuring multi-touch interactions techniques. We demonstrate the possibility to describe touch interaction techniques in a complete and unambiguous way and that the formal description technique is amenable to verification. The capabilities of the notation is demonstrated over two different interaction techniques (namely Pitch and Tap and Hold) together with a software architecture explaining how these interaction techniques can be embedded in an interactive application.

[1]  Bill Buxton,et al.  Multi-Touch Systems that I Have Known and Loved , 2009 .

[2]  Kosuke Tatsukawa,et al.  Graphical toolkit approach to user interaction description , 1991, CHI.

[3]  Tsuneo Nitta,et al.  XISL: a language for describing multimodal interaction scenarios , 2003, ICMI '03.

[4]  Michael D. Harrison,et al.  Prototyping Pre-implementation Designs of Virtual Environment Behaviour , 2001, EHCI.

[5]  Michael D. Harrison,et al.  Formal analysis of ubiquitous computing environments through the APEX framework , 2012, EICS '12.

[6]  Shumin Zhai,et al.  High precision touch screen interaction , 2003, CHI '03.

[7]  Luca Cardelli,et al.  Squeak: a language for communicating with mice , 1985, SIGGRAPH.

[8]  Chiew-Lan Tai,et al.  Multitouch finger registration and its applications , 2010, OZCHI '10.

[9]  Philippe A. Palanque,et al.  Validating interactive system design through the verification of formal task and system models , 1995, EHCI.

[10]  Lars Michael Kristensen,et al.  Coloured Petri Nets and CPN Tools for modelling and validation of concurrent systems , 2007, International Journal on Software Tools for Technology Transfer.

[11]  Robin P. Fawcett,et al.  Theory and application , 1988 .

[12]  John L. Sibert,et al.  An object-oriented user interface management system , 1986, SIGGRAPH.

[13]  Marco Winckler,et al.  A model-based approach for supporting engineering usability evaluation of interaction techniques , 2011, EICS '11.

[14]  Brad A. Myers,et al.  ConstraintJS: programming interactive behaviors for the web by integrating constraints and states , 2012, UIST '12.

[15]  Padmanabhan Krishnan,et al.  Comparing model checking and static program analysis: A case study in error detection approaches , 2010 .

[16]  Marco Winckler,et al.  Beyond modelling: an integrated environment supporting co-execution of tasks and systems models , 2010, EICS '10.

[17]  Philippe A. Palanque,et al.  A model-based tool for interactive prototyping of highly interactive applications , 2002, CHI Extended Abstracts.

[18]  Tony DeRose,et al.  Eden: a professional multitouch tool for constructing virtual organic environments , 2011, CHI.

[19]  Mike Sinclair,et al.  Interaction and modeling techniques for desktop two-handed input , 1998, UIST '98.

[20]  Philip R. Cohen,et al.  QuickSet: multimodal interaction for distributed applications , 1997, MULTIMEDIA '97.

[21]  Caroline Appert,et al.  SwingStates: adding state machines to the swing toolkit , 2006, UIST.

[22]  Karin Coninx,et al.  Integrating Support for Usability Evaluation into High Level Interaction Descriptions with NiMMiT , 2006, DSV-IS.

[23]  Pierre Dragicevic,et al.  Support for input adaptability in the ICON toolkit , 2004, ICMI '04.

[24]  Brad A. Myers,et al.  A user interface toolkit based on graphical objects and constraints , 1988, OOPSLA '88.

[25]  Peter H. Starke,et al.  Analyse von Petri-Netz-Modellen , 1990, Leitfäden und Monographien der Informatik.

[26]  Patrick Baudisch,et al.  Precise selection techniques for multi-touch screens , 2006, CHI.

[27]  Tony DeRose,et al.  Proton++: a customizable declarative multitouch framework , 2012, UIST.

[28]  Eric Barboni,et al.  ICOs: A model-based user interface description technique dedicated to interactive systems addressing usability, reliability and scalability , 2009, TCHI.

[29]  Philippe A. Palanque,et al.  Fusion engines for multimodal input: a survey , 2009, ICMI-MLMI '09.

[30]  Eric Barboni,et al.  High-Fidelity Prototyping of Interactive Systems Can Be Formal Too , 2009, HCI.

[31]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[32]  Raimund Dachselt,et al.  Investigating multi-touch and pen gestures for diagram editing on interactive surfaces , 2009, ITS '09.

[33]  Ted G. Lewis,et al.  Direct-manipulation user interface modeling with high-level Petri nets , 1991, CSC '91.

[34]  Songyang Lao,et al.  A gestural interaction design model for multi-touch displays , 2009, BCS HCI.

[35]  Renaud Blanch,et al.  Programming rich interactions using the hierarchical state machine toolkit , 2006, AVI '06.

[36]  D. Kammer,et al.  Taxonomy and Overview of Multi-touch Frameworks: Architecture, Scope and Features , 2010 .

[37]  Eric Barboni,et al.  Model-Based Engineering of Widgets, User Applications and Servers Compliant with ARINC 661 Specification , 2006, DSV-IS.

[38]  Gudrun Klinker,et al.  A multitouch software architecture , 2008, NordiCHI.

[39]  Rainer Groh,et al.  Towards a formalization of multi-touch gestures , 2010, ITS '10.

[40]  Philippe A. Palanque,et al.  A Formal Approach for User Interaction Reconfiguration of Safety Critical Interactive Systems , 2008, SAFECOMP.

[41]  Benjamin Michotte,et al.  USIXML: A Language Supporting Multi-path Development of User Interfaces , 2004, EHCI/DS-VIS.

[42]  Philippe A. Palanque,et al.  Petri net objects for the design, validation and prototyping of user-driven interfaces , 1990, INTERACT.

[43]  Marc Erich Latoschik Designing transition networks for multimodal VR-interactions using a markup language , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.