A specification paradigm for the design and implementation of tangible user interfaces

Tangible interaction shows promise to significantly enhance computer-mediated support for activities such as learning, problem solving, and design. However, tangible user interfaces are currently considered challenging to design and build. Designers and developers of these interfaces encounter several conceptual, methodological, and technical difficulties. Among others, these challenges include: the lack of appropriate interaction abstractions, the shortcomings of current user interface software tools to address continuous and parallel interactions, as well as the excessive effort required to integrate novel input and output technologies. To address these challenges, we propose a specification paradigm for designing and implementing Tangible User Interfaces (TUIs), that enables TUI developers to specify the structure and behavior of a tangible user interface using high-level constructs which abstract away implementation details. An important benefit of this approach, which is based on User Interface Description Language (UIDL) research, is that these specifications could be automatically or semi-automatically converted into concrete TUI implementations. In addition, such specifications could serve as a common ground for investigating both design and implementation concerns by TUI developers from different disciplines. Thus, the primary contribution of this article is a high-level UIDL that provides developers from different disciplines means for effectively specifying, discussing, and programming a broad range of tangible user interfaces. There are three distinct elements to this contribution: a visual specification technique that is based on Statecharts and Petri nets, an XML-compliant language that extends this visual specification technique, as well as a proof-of-concept prototype of a Tangible User Interface Management System (TUIMS) that semi-automatically translates high-level specifications into a program controlling specific target technologies.

[1]  C. Petri.  Kommunikation mit Automaten , 1962 .

[2]  William M. Newman,et al.  A system for interactive graphical programming , 1968, AFIPS Spring Joint Computing Conference.

[3]  J. D. Foley,et al.  The art of natural graphic man-machine conversation , 1974, SIGGRAPH 1974.

[4]  James D. Foley,et al.  The art of natural graphic man—Machine conversation , 1974 .

[5]  Dan R. Olsen,et al.  SYNGRAPH: A graphical user interface generator , 1983, SIGGRAPH.

[6]  Dan R. Olsen,et al.  Pushdown automata for user interface management , 1984, TOGS.

[7]  Robert J. K. Jacob,et al.  A Specification Language for Direct-Manipulation User Interfaces , 1986, ACM Trans. Graph..

[8]  Wladyslaw M. Turski,et al.  No Silver Bullet - Essence and Accidents of Software Engineering - Response , 1986, IFIP Congress.

[9]  K Sloan,et al.  The hand that rocks the cradle. , 1987, Film History.

[10]  Frederick P. Brooks,et al.  No Silver Bullet: Essence and Accidents of Software Engineering , 1987 .

[11]  David Harel,et al.  On visual formalisms , 1988, CACM.

[12]  G. W. Teuscher.  The hand that rocks the cradle... , 1988, ASDC journal of dentistry for children.

[13]  Deborah Hix,et al.  Human-computer interface development: concepts and systems for its management , 1989, CSUR.

[14]  Philippe A. Palanque,et al.  Petri net objects for the design, validation and prototyping of user-driven interfaces , 1990, INTERACT.

[15]  Dan R. Olsen,et al.  User Interface Management Systems: Models and Algorithms , 1991 .

[16]  Nasa,et al.  A metamodel for the runtime architecture of an interactive system: the UIMS tool developers workshop , 1992, SGCH.

[17]  Jürgen Ziegler,et al.  Generating user interfaces from data models and dialogue net specifications , 1993, INTERCHI.

[18]  R. Polynor,et al.  The hand that rocks the cradle , 1995 .

[19]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[20]  Dan R. Olsen,et al.  Developing user interfaces , 1998 .

[21]  Danny Goodman,et al.  The complete HyperCard handbook , 1998 .

[22]  Robert J. K. Jacob,et al.  A software model and specification language for non-WIMP user interfaces , 1999, TCHI.

[23]  H. Ishii,et al.  Urp: a luminous-tangible workbench for urban planning and design , 1999, CHI '99.

[24]  Brad A. Myers,et al.  Past, Present and Future of User Interface Software Tools , 2000, TCHI.

[25]  Joëlle Coutaz,et al.  A Unifying Reference Framework for the Development of Plastic User Interfaces , 2001, EHCI.

[26]  Saul Greenberg,et al.  Phidgets: easy development of physical interfaces through physical widgets , 2001, UIST '01.

[27]  Mark W. Newman,et al.  The designers' outpost: a tangible interface for collaborative web site , 2001, UIST '01.

[28]  Marc Abrams,et al.  Building Multi-Platform User Interfaces with UIML , 2001, CADUI.

[29]  Fabio Paternò,et al.  One Model, Many Interfaces , 2002, CADUI.

[30]  Ellen Yi-Luen Do,et al.  Navigational blocks: navigating information space with tangible media , 2002, IUI '02.

[31]  Brygg Ullmer,et al.  Tangible interfaces for manipulating aggregates of digital information , 2002 .

[32]  H. James Hoover,et al.  InTml: a description language for VR applications , 2002, Web3D '02.

[33]  Hiroshi Ishii,et al.  A tangible interface for organizing information using a grid , 2002, CHI.

[34]  Ekkart Kindler,et al.  The Petri Net Markup Language , 2003, Petri Net Technology for Communication-Based Systems.

[35]  Meredith Ringel Morris,et al.  iStuff: a physical user interface toolkit for ubiquitous computing environments , 2003, CHI '03.

[36]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[37]  Benjamin Michotte,et al.  USIXML: A Language Supporting Multi-path Development of User Interfaces , 2004, EHCI/DS-VIS.

[38]  Jean Vanderdonckt,et al.  Developing User Interfaces with XML: Advances on User Interface Description Language , 2004 .

[39]  Orit Shaer,et al.  The TAC paradigm: specifying tangible user interfaces , 2004, Personal and Ubiquitous Computing.

[40]  Scott R. Klemmer,et al.  Papier-Mache: toolkit support for tangible input , 2004, CHI.

[41]  Alan F. Blackwell,et al.  Using solid diagrams for tangible interface prototyping , 2005, IEEE Pervasive Comput..

[42]  Michael S. Bernstein,et al.  Reflective physical prototyping through integrated design, test, and analysis , 2006, UIST.

[43]  Jacob Buur,et al.  Getting a grip on tangible interaction: a framework on physical space and social interaction , 2006, CHI.

[44]  Gregory D. Abowd,et al.  Storyboarding: an empirical determination of best practices and effective guidelines , 2006, DIS '06.

[45]  Caroline Appert,et al.  SwingStates: adding state machines to the swing toolkit , 2006, UIST.

[46]  Michael S. Horn,et al.  Tangible programming in the classroom with tern , 2007, CHI Extended Abstracts.

[47]  Scott R. Klemmer,et al.  Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition , 2007, CHI.

[48]  Orit Shaer,et al.  The tangible video editor: collaborative video editing with active tokens , 2007, Tangible and Embedded Interaction.

[49]  Joëlle Coutaz,et al.  BUILDING USER INTERFACES: ORGANIZING SOFTWARE AGENTS , 2007 .

[50]  Doug A. Bowman,et al.  Tiered Developer-Centric Representations for 3D Interfaces: Concept-Oriented Design in Chasm , 2008, 2008 IEEE Virtual Reality Conference.

[51]  Orit Shaer,et al.  Marble track audio manipulator (MTAM): a tangible user interface for audio composition , 2008, Tangible and Embedded Interaction.

[52]  Orit Shaer,et al.  Reality-based interaction: a framework for post-WIMP interfaces , 2008, CHI.

[53]  K. Gegenfurtner,et al.  Design Issues in Gaze Guidance Under review with ACM Transactions on Computer Human Interaction , 2009 .