High-Level Modeling of Multimodal Interaction Techniques Using NiMMiT

JVRB, 4(2007), no. 2. - The past few years, multimodal interaction has been gaining importance in virtual environments. Although multimodality renders interacting with an environment more natural and intuitive, the development cycle of such an application is often long and expensive. In our overall field of research, we investigate how modelbased design can facilitate the development process by designing environments through the use of highlevel diagrams. In this scope, we present ‘NiMMiT’, a graphical notation for expressing and evaluating multimodal user interaction; we elaborate on the NiMMiT primitives and demonstrate its use by means of a comprehensive example.

[1]  Marisa E. Campbell,et al.  Conference preview: HCI 2000: usability or else! , 2000, INTR.

[2]  H. James Hoover,et al.  InTml: a description language for VR applications , 2002, Web3D '02.

[3]  Joëlle Coutaz,et al.  A generic platform for addressing the multimodal challenge , 1995, CHI '95.

[4]  Kris Luyten,et al.  DynaMo-AID: A Design Process and a Runtime Architecture for Dynamic Model-Based User Interface Development , 2004, EHCI/DS-VIS.

[5]  Doug A. Bowman,et al.  An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments , 1997, SI3D.

[6]  Karin Coninx,et al.  Integrating Support for Usability Evaluation into High Level Interaction Descriptions with NiMMiT , 2006, DSV-IS.

[7]  Pierre Dragicevic,et al.  The MaggLite post-WIMP toolkit: draw it, connect it and run it , 2004, UIST '04.

[8]  David A. Carr,et al.  Interaction object graphs : an executable graphical notation for specifying user interfaces , 1997 .

[9]  Marti A. Hearst,et al.  The state of the art in automating usability evaluation of user interfaces , 2001, CSUR.

[10]  Mark Green,et al.  JDCAD: A highly interactive 3D modeling system , 1994, Comput. Graph..

[11]  Karin Coninx,et al.  A Model-Based Design Process for Interactive Virtual Environments , 2005, DSV-IS.

[12]  David Harel,et al.  Statecharts: A Visual Formalism for Complex Systems , 1987, Sci. Comput. Program..

[13]  Rüdiger Valk,et al.  Petri Nets as Token Objects: An Introduction to Elementary Object Nets , 1998, ICATPN.

[14]  Karin Coninx,et al.  Multisensory interaction metaphors with haptics and proprioception in virtual environments , 2004, NordiCHI '04.

[15]  Karin Coninx,et al.  Are existing metaphors in virtual environments suitable for haptic interaction , 2005 .

[16]  Pierre Dragicevic,et al.  Support for input adaptability in the ICON toolkit , 2004, ICMI '04.

[17]  Paulo Pinheiro,et al.  User Interface Declarative Models and Development Environments: A Survey , 2000, DSV-IS.

[18]  C. A. Petri.  Fundamentals of a Theory of Asynchronous Information Flow , 1962, IFIP Congress.

[19]  Ann Blandford,et al.  Four easy pieces for assessing the usability of multimodal interaction: the CARE properties , 1995, INTERACT.

[20]  Roberto Ierusalimschy,et al.  Lua—An Extensible Extension Language , 1996 .

[21]  Kurt Jensen,et al.  An Introduction to the Theoretical Aspects of Coloured Petri Nets , 1993, REX School/Symposium.

[22]  Marco Winckler,et al.  A Formal Description of Multimodal Interaction Techniques for Immersive Virtual Reality Applications , 2005, INTERACT.

[23]  Karin Coninx,et al.  Comparing NiMMiT and Data-Driven Notations for Describing Multimodal Interaction , 2006, TAMODIA.

[24]  Philippe A. Palanque,et al.  Petri net based Design of User-driven Interfaces Using the Interactive Cooperative Objects Formalism , 1994, DSV-IS.

[25]  Janienke Sturm,et al.  The effect of prolonged use on multimodal interaction , 2002 .

[26]  Randy F. Pausch,et al.  Voodoo dolls: seamless interaction at multiple scales in virtual environments , 1999, SI3D.