Flexpad: highly flexible bending interactions for projected handheld displays

Flexpad is an interactive system that combines a depth camera and a projector to transform sheets of plain paper or foam into flexible, highly deformable, and spatially aware handheld displays. We present a novel approach for tracking deformed surfaces from depth images in real time. It captures deformations in high detail, is very robust to occlusions created by the user's hands and fingers, and does not require any kind of markers or visible texture. As a result, the display is considerably more deformable than in previous work on flexible handheld displays, enabling novel applications that leverage the high expressiveness of detailed deformation. We illustrate these unique capabilities through three application examples: curved cross-cuts in volumetric images, deforming virtual paper characters, and slicing through time in videos. Results from two user studies show that our system is capable of detecting complex deformations and that users are able to perform them quickly and precisely.

[1]  George W. Fitzmaurice,et al.  Exploring interactive curve and surface manipulation using a bend and twist sensitive input strip , 1999, SI3D.

[2]  N. Hansen,et al.  An Evolution Strategy with Coordinate System Invariant Adaptation of Arbitrary Normal Mutation Distr , 1999 .

[3]  Ramesh Raskar,et al.  Dynamic shader lamps : painting on movable objects , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[4]  R. E. Mark,et al.  Handbook of physical testing of paper , 2002 .

[5]  Hiroshi Ishii,et al.  Illuminating clay: a 3-D tangible interface for landscape analysis , 2002, CHI.

[6]  Roger D. Hersch,et al.  Exploring curved anatomic structures with surface sections , 2003, IEEE Visualization, 2003. VIS 2003..

[7]  Ivan Poupyrev,et al.  Gummi: a bendable computer , 2004, CHI '04.

[8]  Gary W. Meyer,et al.  A handheld flexible display system , 2005, VIS 05. IEEE Visualization, 2005..

[9]  Nikolaus F. Troje,et al.  Paper windows: interaction techniques for digital paper , 2005, CHI.

[10]  Masatoshi Ishikawa,et al.  Khronos projector , 2005, SIGGRAPH '05.

[11]  Thomas S. Huang,et al.  Multicue HMM-UKF for real-time contour tracking , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Mark Billinghurst,et al.  A 3D Flexible and Tangible Magic Lens in Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[13]  Marc Alexa,et al.  As-rigid-as-possible surface modeling , 2007, Symposium on Geometry Processing.

[14]  Roel Vertegaal,et al.  Towards more paper-like input: flexible input devices for foldable interaction styles , 2008, UIST '08.

[15]  Jan O. Borchers,et al.  Twend: twisting and bending as new interaction gesture in mobile devices , 2008, CHI Extended Abstracts.

[16]  Scott E. Hudson,et al.  Foldable interactive displays , 2008, UIST '08.

[17]  Dan B. Goldman,et al.  Video puppetry: a performative interface for cutout animation , 2008, SIGGRAPH Asia '08.

[18]  Raimund Dachselt,et al.  PaperLens: advanced magic lens interaction above the tabletop , 2009, ITS '09.

[19]  Pattie Maes,et al.  WUW - wear Ur world: a wearable gestural interface , 2009, CHI Extended Abstracts.

[20]  Pascal Fua,et al.  Template-free monocular reconstruction of deformable surfaces , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[21]  Anna Hilsmann,et al.  Realistic cloth augmentation in single view video under occlusions , 2010, Comput. Graph..

[22]  Slobodan Ilic,et al.  Probabilistic Deformable Surface Tracking from Multiple Videos , 2010, ECCV.

[23]  Xu Jia,et al.  How users manipulate deformable displays as input devices , 2010, CHI.

[24]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[25]  Roel Vertegaal,et al.  PaperPhone: understanding the use of bend gestures in mobile devices with flexible electronic paper displays , 2011, CHI.

[26]  Roel Vertegaal,et al.  Evaluating effects of structural holds on pointing and dragging performance with flexible displays , 2011, CHI.

[27]  Chris Harrison,et al.  OmniTouch: wearable multitouch interaction everywhere , 2011, UIST.

[28]  Max Mühlhäuser,et al.  Xpaaand: interaction techniques for rollable displays , 2011, CHI.

[29]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[30]  Graham A. Wilson,et al.  Feeling it: the roles of stiffness, deformation range and feedback in the control of deformable ui , 2012, ICMI '12.

[31]  Jürgen Steimle,et al.  FoldMe: interacting with double-sided foldable displays , 2012, Tangible and Embedded Interaction.

[32]  Andrew Wilson,et al.  MirageTable: freehand interaction on a projected augmented reality tabletop , 2012, CHI.

[33]  Sriram Subramanian,et al.  Tilt displays: designing display surfaces with multi-axis tilting and actuation , 2012, Mobile HCI.

[34]  Reinhard Koch,et al.  Direct Model-Based Tracking of 3D Object Deformations in Depth and Color Video , 2012, International Journal of Computer Vision.

[35]  David Kim,et al.  HoloDesk: direct 3d interactions with a situated see-through display , 2012, CHI.

[36]  Raimund Dachselt,et al.  Going beyond the surface: studying multi-layer interaction above the tabletop , 2012, CHI.

[37]  Susanna Paasovaara,et al.  Kinetic device: designing interactions with a deformable mobile interface , 2012, CHI Extended Abstracts.

[38]  S. Gehring.  Media Façades : Turning buildings into large-scale interactive surfaces , 2013 .