Towards Around-Device Interaction using Corneal Imaging

Around-device interaction techniques aim at extending the input space using various sensing modalities on mobile and wearable devices. In this paper, we present our work towards extending the input area of mobile devices using front-facing device-centered cameras that capture reflections in the human eye. As current generation mobile devices lack high resolution front-facing cameras we study the feasibility of around-device interaction using corneal reflective imaging based on a high resolution camera. We present a workflow, a technical prototype and an evaluation, including a migration path from high resolution to low resolution imagers. Our study indicates, that under optimal conditions a spatial sensing resolution of 5 cm in the vicinity of a mobile phone is possible.

[1]  Florian Alt,et al.  Challenges and design space of gaze-enabled public displays , 2016, UbiComp Adjunct.

[2]  Atsushi Nakazawa,et al.  I See What You See: Point of Gaze Estimation from Corneal Images , 2013, 2013 2nd IAPR Asian Conference on Pattern Recognition.

[3]  Atsushi Nakazawa,et al.  Corneal Imaging Revisited: An Overview of Corneal Reflection Analysis and Applications , 2013, IPSJ Trans. Comput. Vis. Appl..

[4]  J. Snyder.  Flattening the Earth: Two Thousand Years of Map Projections , 1994 .

[5]  Richard Martin,et al.  Design for wearability , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[6]  Ian Oakley,et al.  Interaction on the edge: offset sensing for small devices , 2014, CHI.

[7]  Xiang 'Anthony' Chen,et al.  Duet: exploring joint interactions on a smart phone and a smart watch , 2014, CHI.

[8]  Shree K. Nayar,et al.  Corneal Imaging System: Environment from Eyes , 2006, International Journal of Computer Vision.

[9]  Shwetak N. Patel,et al.  SideSwipe: detecting in-air gestures around mobile devices using actual GSM signal , 2014, UIST.

[10]  Peter Robinson,et al.  A 3D Morphable Eye Region Model for Gaze Estimation , 2016, ECCV.

[11]  Stephen Lin,et al.  Separating corneal reflections for illumination estimation , 2008, Neurocomputing.

[12]  Hans-Werner Gellersen,et al.  Orbits: enabling gaze interaction in smart watches using moving targets , 2015, UbiComp/ISWC Adjunct.

[13]  Otmar Hilliges,et al.  In-air gestures around unmodified mobile devices , 2014, UIST.

[14]  Mark Rosenfield,et al.  Font Size and Viewing Distance of Handheld Smart Phones , 2011, Optometry and vision science : official publication of the American Academy of Optometry.

[15]  Robert Xiao,et al.  Toffee: enabling ad hoc, around-device interaction with acoustic time-of-arrival correlation , 2014, MobileHCI '14.

[16]  Shahram Izadi,et al.  SideSight: multi-"touch" interaction around small devices , 2008, UIST '08.

[17]  Yvonne Rogers,et al.  HuddleLamp: Spatially-Aware Mobile Displays for Ad-hoc Around-the-Table Collaboration , 2014, ITS '14.

[18]  Michael Rohs,et al.  Hoverflow: exploring around-device interaction with IR distance sensors , 2009, Mobile HCI.

[19]  Da-Yuan Huang,et al.  Cyclops: Wearable and Single-Piece Full-Body Gesture Input Devices , 2015, CHI.

[20]  Mario Fritz,et al.  It’s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Xing-Dong Yang,et al.  Surround-see: enabling peripheral vision on smartphones during active use , 2013, UIST.

[22]  Martin Welk,et al.  Tempest in a Teapot: Compromising Reflections Revisited , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[23]  Desney S. Tan,et al.  FingerIO: Using Active Sonar for Fine-Grained Finger Tracking , 2016, CHI.

[24]  Yvonne Rogers,et al.  Fat Finger Worries: How Older and Younger Users Physically Interact with PDAs , 2005, INTERACT.

[25]  Dominik Schmidt,et al.  A cross-device interaction style for mobiles and surfaces , 2012, DIS '12.

[26]  Zehdreh Allen-Lafayette,et al.  Flattening the Earth, Two Thousand Years of Map Projections , 1998 .

[27]  Marc Langheinrich,et al.  Back-of-device authentication on smartphones , 2013, CHI.

[28]  Shree K. Nayar,et al.  Eyes for relighting , 2004, ACM Trans. Graph..

[29]  Dieter Schmalstieg,et al.  GlassHands: Interaction Around Unmodified Mobile Devices Using Sunglasses , 2016, ISS.

[30]  Mohamed Rizon,et al.  Detection of eyes from human faces by Hough transform and separability filter , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[31]  Patrick Baudisch,et al.  Lucid touch: a see-through mobile device , 2007, UIST.

[32]  Kwan-Yee Kenneth Wong,et al.  Reconstruction of display and eyes from a single image , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[33]  Andreas Bulling,et al.  EyeTab: model-based gaze estimation on unmodified tablet computers , 2014, ETRA.

[34]  Dieter Schmalstieg,et al.  MultiFi: Multi Fidelity Interaction with Displays On and Around the Body , 2015, CHI.

[35]  Stephen A. Brewster,et al.  Towards usable and acceptable above-device interactions , 2014, MobileHCI '14.

[36]  Florian Alt,et al.  GazeTouchPass: Multimodal Authentication Using Gaze and Touch on Mobile Devices , 2016, CHI Extended Abstracts.

[37]  Ravi Kothari,et al.  Detection of eye locations in unconstrained visual images , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[38]  Atsushi Nakazawa,et al.  Display-camera calibration using eye reflections and geometry constraints , 2011, Comput. Vis. Image Underst..

[39]  Chris Harrison.  Appropriated Interaction Surfaces , 2010, Computer.