TouchTools: leveraging familiarity and skill with physical tools to augment touch interaction

The average person can skillfully manipulate a plethora of tools, from hammers to tweezers. However, despite this remarkable dexterity, gestures on today's touch devices are simplistic, relying primarily on the chording of fingers: one-finger pan, two-finger pinch, four-finger swipe and similar. We propose that touch gesture design be inspired by the manipulation of physical tools from the real world. In this way, we can leverage user familiarity and fluency with such tools to build a rich set of gestures for touch interaction. With only a few minutes of training on a proof-of-concept system, users were able to summon a variety of virtual tools by replicating their corresponding real-world grasps.

[1]  Burnett William.  Clinical Science for Surgeons , 1981 .

[2]  F. R. Wilson.  The Hand: How Its Use Shapes the Brain, Language, and Human Culture , 1998 .

[3]  D. Evans,et al.  Rehabilitation of the hand , 1973 .

[4]  Daniel J. Wigdor,et al.  Rock & rails: extending multi-touch interactions with shape gestures to enable precise spatial manipulations , 2011, CHI.

[5]  Amy Wohl,et al.  Apple computer, Inc. , 2003 .

[6]  Dominik Schmidt,et al.  HandsDown: hand-contour-based user identification for interactive surfaces , 2010, NordiCHI.

[7]  Mike Wu,et al.  Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays , 2003, UIST '03.

[8]  Patrick Olivier,et al.  SurfaceMouse: supplementing multi-touch interaction with a virtual mouse , 2011, Tangible and Embedded Interaction.

[9]  James D. Hollan,et al.  SLAP widgets: bridging the gap between virtual and physical controls on tabletops , 2009, CHI.

[10]  Simon Rogers,et al.  FingerCloud: uncertainty and autonomy handover incapacitive sensing , 2010, CHI.

[11]  Mike Wu,et al.  A study of hand shape use in tabletop gesture interaction , 2006, CHI Extended Abstracts.

[12]  Mike Wu,et al.  Gesture registration, relaxation, and reuse for multi-point direct-touch surfaces , 2006, First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06).

[13]  Michel Beaudouin-Lafon,et al.  Charade: remote control of objects using free-hand gestures , 1993, CACM.

[14]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.

[15]  Andrew D. Wilson.  Simulating grasping behavior on an imaging interactive surface , 2009, ITS '09.

[16]  C. Parry.  Rehabilitation of the hand , 1973 .

[17]  Otmar Hilliges,et al.  Bringing physics to the surface , 2008, UIST '08.

[18]  Xiang Cao,et al.  ShapeTouch: Leveraging contact shape on interactive surfaces , 2008, 2008 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems.

[19]  Michael Rohs,et al.  PalmSpace: continuous around-device gestures vs. multitouch for 3D rotation tasks on mobile devices , 2012, AVI.

[20]  Bernd Fröhlich,et al.  Finger and hand detection for multi-touch interfaces based on maximally stable extremal regions , 2012, ITS '12.

[21]  Meredith Ringel Morris,et al.  ShadowGuides: visualizations for in-situ learning of multi-touch and whole-hand gestures , 2009, ITS '09.

[22]  Sebastian Boring,et al.  Designing user-, hand-, and handpart-aware tabletop interactions with the TouchID toolkit , 2011, ITS '11.

[23]  Ivan Poupyrev,et al.  A framework and testbed for studying manipulation techniques for immersive VR , 1997, VRST '97.

[24]  William Buxton,et al.  Pen + touch = new tools , 2010, UIST.

[25]  Karthik Ramani,et al.  Extended multitouch: recovering touch posture and differentiating users using a depth camera , 2012, UIST.

[26]  Christine L. MacKenzie,et al.  The Grasping Hand , 2011, The Grasping Hand.

[27]  Mircea Nicolescu,et al.  Vision-based hand pose estimation: A review , 2007, Comput. Vis. Image Underst..

[28]  Patrick Baudisch,et al.  The generalized perceived input point model and how to double touch accuracy by extracting fingerprints , 2010, CHI.

[29]  G. H. Anderson.  Advances in Clinical Cytology. , 1982 .

[30]  Jun Rekimoto,et al.  SmartSkin: an infrastructure for freehand manipulation on interactive surfaces , 2002, CHI.