dSensingNI: a framework for advanced tangible interaction using a depth camera

This work presents an approach to overcome the disadvantages of existing interaction frameworks and technologies for touch detection and object interaction. The robust and easy to use framework dSensingNI (Depth Sensing Natural Interaction) is described, which supports multitouch and tangible interaction with arbitrary objects. It uses images from a depth-sensing camera and provides tracking of users fingers of palm of hands and combines this with object interaction, such as grasping, grouping and stacking, which can be used for advanced interaction techniques.

[1]  Daisuke Sakamoto,et al.  CRISTAL: a collaborative home media and device controller based on a multi-touch display , 2009, ITS '09.

[2]  Patrick Baudisch,et al.  Lumino: tangible blocks for tabletop computers based on glass fiber bundles , 2010, CHI.

[3]  Eva Hornecker,et al.  Tangible user interfaces als kooperationsunterstützendes Medium , 2004 .

[4]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[5]  Andrew D. Wilson.  Using a depth camera as a touch sensor , 2010, ITS '10.

[6]  Andrew D. Wilson.  PlayAnywhere: a compact interactive tabletop projection-vision system , 2005, UIST.

[7]  Karsten Nebe,et al.  An Investigation on Requirements for Co-located Group-Work Using Multitouch-, Pen-Based- and Tangible-Interaction , 2011, HCI.

[8]  Sergi Jordà,et al.  The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces , 2007, TEI.

[9]  Ayoub Al-Hamadi,et al.  Data Gathering for Gesture Recognition Systems Based on Mono Color-, Stereo Color- and Thermal Cameras , 2009, FGIT.

[10]  Andreas Butz,et al.  Interactions in the air: adding further depth to interactive tabletops , 2009, UIST '09.

[11]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[12]  Michael Haller,et al.  Geckos: combining magnets and pressure images to enable new tangible-object design and interaction , 2011, CHI.

[13]  Max Mühlhäuser,et al.  Workshop on interacting with smart objects , 2011, IUI '11.

[14]  Daniel C. Robbins,et al.  PlayTogether: Playing Games across Multiple Interactive Tabletops , 2006 .

[15]  Tom Bartindale,et al.  Stacks on the surface: resolving physical order using fiducial markers with structured transparency , 2009, ITS '09.

[16]  Regan L. Mandryk,et al.  Direct intentions: the effects of input devices on collaboration around a tabletop display , 2006, First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06).

[17]  Robert Aish.  3D input for CAAD systems , 1979 .

[18]  Brian P. Bailey,et al.  Build your world and play in it: Interacting with surface particles on complex objects , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[19]  Y Wang,et al.  Bringing Clay and Sand into Digital Design — Continuous Tangible user Interfaces , 2004 .

[20]  Hrvoje Benko,et al.  Beyond flat surface computing: challenges of depth-aware and curved interfaces , 2009, ACM Multimedia.

[21]  James D. Hollan,et al.  SLAP widgets: bridging the gap between virtual and physical controls on tabletops , 2009, CHI.

[22]  Thomas Seifried,et al.  CRISTAL : Design and Implementation of a Remote Control System Based on a Multi-touch Display , 2009 .

[23]  Luc Van Gool,et al.  Tracking a hand manipulating an object , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[24]  Chris Harrison,et al.  OmniTouch: wearable multitouch interaction everywhere , 2011, UIST.

[25]  Enrico Costanza,et al.  TUIO: A Protocol for Table-Top Tangible User Interfaces , 2005 .