GlassHands: Interaction Around Unmodified Mobile Devices Using Sunglasses

We present a novel approach for extending the input space around unmodified mobile devices. Using built-in front-facing cameras of unmodified handheld devices, GlassHands estimates hand poses and gestures through reflections in sunglasses, ski goggles or visors. Thereby, GlassHands creates an enlarged input space, rivaling input reach on large touch displays. We introduce the idea along with its technical concept and implementation. We demonstrate the feasibility and potential of our proposed approach in several application scenarios, such as map browsing or drawing using a set of interaction techniques previously possible only with modified mobile devices or on large touch displays. Our research is backed up with a user study.

[1]  Shahram Izadi,et al.  SideSight: multi-"touch" interaction around small devices , 2008, UIST '08.

[2]  Otmar Hilliges,et al.  In-air gestures around unmodified mobile devices , 2014, UIST.

[3]  Sean White,et al.  Nenya: subtle and eyes-free mobile input with a magnetically-tracked finger ring , 2011, CHI.

[4]  Chris Harrison,et al.  OmniTouch: wearable multitouch interaction everywhere , 2011, UIST.

[5]  Saul Greenberg,et al.  The Continuous Interaction Space: Interaction Techniques Unifying Touch and Gesture on and above a Digital Surface , 2011, INTERACT.

[6]  Tong Lu,et al.  iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing , 2015, CHI.

[7]  David A. Forsyth,et al.  BeThere: 3D mobile collaboration with spatial input , 2013, CHI.

[8]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Wendy E. Mackay,et al.  Body-centric design space for multi-surface interaction , 2013, CHI.

[10]  Jian Sun,et al.  Face Alignment at 3000 FPS via Regressing Local Binary Features , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Stephen A. Brewster,et al.  Towards usable and acceptable above-device interactions , 2014, MobileHCI '14.

[12]  Jacob O. Wobbrock,et al.  Portico: tangible interaction on and around a tablet , 2011, UIST.

[13]  Robert Xiao,et al.  Toffee: enabling ad hoc, around-device interaction with acoustic time-of-arrival correlation , 2014, MobileHCI '14.

[14]  Nassir Navab,et al.  Visual marker detection and decoding in AR systems: a comparative study , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[15]  Fabrice Matulic,et al.  Sensing techniques for tablet+stylus interaction , 2014, UIST.

[16]  Michael Rohs,et al.  PalmSpace: continuous around-device gestures vs. multitouch for 3D rotation tasks on mobile devices , 2012, AVI.

[17]  Ian Oakley,et al.  Interaction on the edge: offset sensing for small devices , 2014, CHI.

[18]  Richard Martin,et al.  Design for wearability , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[19]  Geoffrey J. McLachlan,et al.  Mixture models : inference and applications to clustering , 1989 .

[20]  Dieter Schmalstieg,et al.  MultiFi: Multi Fidelity Interaction with Displays On and Around the Body , 2015, CHI.

[21]  Ramesh Raskar,et al.  VRCodes: Unobtrusive and active visual codes for interaction by exploiting rolling shutter , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[22]  Marc Langheinrich,et al.  Back-of-device authentication on smartphones , 2013, CHI.

[23]  Xiang 'Anthony' Chen,et al.  Skin buttons: cheap, small, low-powered and clickable fixed-icon laser projectors , 2014, UIST.

[24]  Xiang 'Anthony' Chen,et al.  Air+touch: interweaving touch & in-air gestures , 2014, UIST.

[25]  Chris Harrison,et al.  Abracadabra: wireless, high-precision, and unpowered finger input for very small mobile devices , 2009, UIST '09.

[26]  Dominik Schmidt,et al.  A cross-device interaction style for mobiles and surfaces , 2012, DIS '12.

[27]  Mircea Nicolescu,et al.  Vision-based hand pose estimation: A review , 2007, Comput. Vis. Image Underst..

[28]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[29]  Simone Gasparini,et al.  Camera Models and Fundamental Concepts Used in Geometric Computer Vision , 2011, Found. Trends Comput. Graph. Vis..

[30]  Michita Imai,et al.  SkinWatch: skin gesture interaction for smart watch , 2015, AH.

[31]  Atsushi Nakazawa,et al.  Corneal Imaging Revisited: An Overview of Corneal Reflection Analysis and Applications , 2013, IPSJ Trans. Comput. Vis. Appl..

[32]  Pourang Irani,et al.  Ad-binning: leveraging around device space for storing, browsing and retrieving mobile device content , 2013, CHI.

[33]  Tovi Grossman,et al.  Medusa: a proximity-aware multi-touch tabletop , 2011, UIST.

[34]  Xiang 'Anthony' Chen,et al.  Duet: exploring joint interactions on a smart phone and a smart watch , 2014, CHI.

[35]  Shwetak N. Patel,et al.  SideSwipe: detecting in-air gestures around mobile devices using actual GSM signal , 2014, UIST.

[36]  David A. Forsyth,et al.  Around device interaction for multiscale navigation , 2012, Mobile HCI.

[37]  Steven K. Feiner,et al.  Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality , 2008, VRST '08.

[38]  Xing-Dong Yang,et al.  Surround-see: enabling peripheral vision on smartphones during active use , 2013, UIST.

[39]  Yvonne Rogers,et al.  HuddleLamp: Spatially-Aware Mobile Displays for Ad-hoc Around-the-Table Collaboration , 2014, ITS '14.

[40]  Michael Rohs,et al.  Hoverflow: exploring around-device interaction with IR distance sensors , 2009, Mobile HCI.

[41]  Da-Yuan Huang,et al.  Cyclops: Wearable and Single-Piece Full-Body Gesture Input Devices , 2015, CHI.

[42]  Chris Harrison.  Appropriated Interaction Surfaces , 2010, Computer.

[43]  Stephen A. Brewster,et al.  Gestures all around us: user differences in social acceptability perceptions of gesture based interfaces , 2009, Mobile HCI.

[44]  Xiang 'Anthony' Chen,et al.  Around-body interaction: sensing & interaction techniques for proprioception-enhanced input with mobile devices , 2014, MobileHCI '14.

[45]  Abigail Sellen,et al.  Toward compound navigation tasks on mobiles via spatial manipulation , 2013, MobileHCI '13.

[46]  Y. Guiard.  Asymmetric division of labor in human skilled bimanual action: the kinematic chain as a model. , 1987, Journal of motor behavior.

[47]  Patrick Baudisch,et al.  Lucid touch: a see-through mobile device , 2007, UIST.