From gesture to action : natural user interfaces

It is time to go beyond the established approaches in user-system interaction (USI). After a serious critic of well established interaction styles we discuss the two known approaches to overcome the obstacles and limitations: (1) [immersive] Virtual Reality (VR), and (2) Augmented Reality (AR). Both design strategies are diametrically opposed: VR enriches the virtual world with real humans, while AR augments the real world with intelligent features. Only with the AR design strategy humans are able to behave as much as possible in a natural way: behavior of humans in the real world with other humans and/or real world objects. Our interest in user centred design let us follow this idea. Based on the fundamental constraints of natural way of interacting we derive a set of recommendations for the next generation of user interfaces: the Natural User Interface (NUI). The concept of NUI is discussed in form of a general framework and in form of several NUI-like applications. Finally we describe the interdisciplinary research topics that must be taken into consideration to come up one day--in the near future--with a welldesigned NUI. 1. Why do we need a new interaction technique?

[1]  Mark S. Sanders,et al.  Human Factors in Engineering and Design , 2016 .

[2]  Wendy E. Mackay,et al.  Computer-Augmented Environments: Back to the Real World - Introduction to the Special Issue. , 1993 .

[3]  Marimuthu Palaniswami,et al.  Pattern recognition by topology free spatio-temporal feature map , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[4]  Venu Govindaraju,et al.  System for reading handwritten documents , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[5]  Nancy J. Cooke,et al.  The effects of emotional icons on remote communication , 1996, CHI Conference Companion.

[6]  Hiroshi Ishii,et al.  Bricks: laying the foundations for graspable user interfaces , 1995, CHI '95.

[7]  Donald A. Norman,et al.  User Centered System Design , 1986 .

[8]  P R Cohen,et al.  The role of voice input for human-machine communication. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[9]  Gregory Kramer,et al.  Auditory Display: Sonification, Audification, And Auditory Interfaces , 1994 .

[10]  R. Exline,et al.  Social visual interaction: A conceptual and literature review. , 1987 .

[11]  Jakob Nielsen.  Book review: Designing the User Interface: Strategies for Effective Human-Computer Interaction by Ben Shneiderman (Addison-Wesley, 1987) , 1987, SGCH.

[12]  Ben Shneiderman,et al.  Designing The User Interface , 2013 .

[13]  G.W.M. Rauterberg,et al.  From competition to collaboration through a shared social space , 1995 .

[14]  N. Ohnishi,et al.  Deformable template based stereo , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[15]  Elizabeth M. Wenzel,et al.  Localization in Virtual Acoustic Displays , 1992, Presence: Teleoperators & Virtual Environments.

[16]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[17]  Thomas Berlage,et al.  REAL-TIME INTEGRATION OF SYNTHETIC COMPUTER GRAPHICS INTO LIVE VIDEO SCENES , 1995 .

[18]  Quentin Stafford-Fraser,et al.  BrightBoard: a video-augmented environment , 1996, CHI '96.

[19]  Pierre David Wellner,et al.  Interacting with paper on the DigitalDesk , 1993, CACM.

[20]  P. Watzlawick,et al.  Pragmatics of Human Communication: A Study of Interactional Patterns, Pathologies and Paradoxes , 1964 .

[21]  Gwm Matthias Rauterberg,et al.  Physical modelling of environmental sounds , 1995 .

[22]  Doug Hill,et al.  Three dimensional reconstruction of trunk surface using structured light , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[23]  Wendy E. Mackay,et al.  Ariel: augmenting paper engineering drawings , 1995, CHI 95 Conference Companion.

[24]  Barry G. Blundell,et al.  Volumetric three-dimensional display systems: their past, present and future , 1993 .

[25]  David B. Pisoni,et al.  Text-to-speech: the mitalk system , 1987 .

[26]  Justine Cassell,et al.  Gesture and Ground , 1990 .

[27]  Morten Fjeld,et al.  BUILD-IT: a planning tool for construction and design , 1998, CHI Conference Summary.

[28]  Victor Zue,et al.  Spoken language systems for human/machine interfaces , 1991, RIAO.

[29]  Bruce Tognazzini.  Tog on Software Design , 1995 .

[30]  Mark Lucente,et al.  Rendering interactive holographic images , 1995, SIGGRAPH.

[31]  Kimiyoshi Machii,et al.  InteractiveDESK: a computer-augmented desk which responds to operations on real objects , 1995, CHI 95 Conference Companion.

[32]  M. Covell.  Autocorrespondence: feature-based match estimation and image metamorphosis , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[33]  Stephen W. Draper,et al.  Display Managers as the Basis for User-Machine Communication , 1986 .

[34]  Wendy E. Mackay,et al.  Augmenting reality: adding computational dimensions to paper , 1993, CACM.