ScreenGlint: Practical, In-situ Gaze Estimation on Smartphones

Gaze estimation has widespread applications. However, little work has explored gaze estimation on smartphones, even though they are fast becoming ubiquitous. This paper presents ScreenGlint, a novel approach which exploits the glint (reflection) of the screen on the user's cornea for gaze estimation, using only the image captured by the front-facing camera. We first conduct a user study on common postures during smartphone use. We then design an experiment to evaluate the accuracy of ScreenGlint under varying face-to-screen distances. An in-depth evaluation involving multiple users is conducted and the impact of head pose variations is investigated. ScreenGlint achieves an overall angular error of 2.44º without head pose variations, and 2.94º with head pose variations. Our technique compares favorably to state-of-the-art research works, indicating that the glint of the screen is an effective and practical cue to gaze estimation on the smartphone platform. We believe that this work can open up new possibilities for practical and ubiquitous gaze-aware applications.

[1]  Atsushi Nakazawa,et al.  Display-camera calibration using eye reflections and geometry constraints , 2011, Comput. Vis. Image Underst..

[2]  Andrew W. Fitzgibbon,et al.  A Buyer's Guide to Conic Fitting , 1995, BMVC.

[3]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[4]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Andrew T. Duchowski,et al.  Eye Tracking Methodology: Theory and Practice , 2003, Springer London.

[6]  Yoichi Sato,et al.  Appearance-Based Gaze Estimation Using Visual Saliency , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Pablo Varona,et al.  Controlling a Smartphone Using Gaze Gestures as the Input Mechanism , 2015, Hum. Comput. Interact..

[8]  Qiang Ji,et al.  In the Eye of the Beholder: A Survey of Models for Eyes and Gaze , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Neil A. Dodgson,et al.  Robust real-time pupil tracking in highly off-axis images , 2012, ETRA.

[10]  Mario Fritz,et al.  Appearance-based gaze estimation in the wild , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Peter Robinson,et al.  Learning an appearance-based gaze estimator from one million synthesised images , 2016, ETRA.

[12]  Takahiro Okabe,et al.  Gaze Estimation From Eye Appearance: A Head Pose-Free Method via Eye Image Synthesis , 2015, IEEE Transactions on Image Processing.

[13]  Wojciech Matusik,et al.  Eye Tracking for Everyone , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Paul A. Viola,et al.  Robust Real-time Object Detection , 2001 .

[15]  Qiong Huang,et al.  TabletGaze: Unconstrained Appearance-based Gaze Estimation in Mobile Tablets , 2015 .

[16]  Hans-Werner Gellersen,et al.  Pursuits: spontaneous interaction with displays based on smooth pursuit eye movement and moving targets , 2013, UbiComp.

[17]  Moshe Eizenman,et al.  General theory of remote gaze estimation using the pupil center and corneal reflections , 2006, IEEE Transactions on Biomedical Engineering.

[18]  Oliver Hohlfeld,et al.  On the Applicability of Computer Vision based Gaze Tracking in Mobile Scenarios , 2015, MobileHCI.

[19]  Kwan-Yee Kenneth Wong,et al.  Reconstruction of display and eyes from a single image , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[20]  Rafael Cabeza,et al.  Models for Gaze Tracking Systems , 2007, EURASIP J. Image Video Process..

[21]  Andreas Bulling,et al.  EyeTab: model-based gaze estimation on unmodified tablet computers , 2014, ETRA.

[22]  Qiong Huang,et al.  TabletGaze: dataset and analysis for unconstrained appearance-based gaze estimation in mobile tablets , 2017, Machine Vision and Applications.

[23]  Qiang Ji,et al.  Deep eye fixation map learning for calibration-free eye gaze tracking , 2016, ETRA.

[24]  Hans-Werner Gellersen,et al.  Orbits: Gaze Interaction for Smart Watches using Smooth Pursuit Eye Movements , 2015, UIST.

[25]  James Hays,et al.  WebGazer: Scalable Webcam Eye Tracking Using User Interactions , 2016, IJCAI.

[26]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[27]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[28]  Soo-Young Lee,et al.  A Study on Human Gaze Estimation Using Screen Reflection , 2008, IDEAL.

[29]  Yanxia Zhang,et al.  Gaussian processes as an alternative to polynomial gaze estimation functions , 2016, ETRA.

[30]  Shumin Zhai,et al.  Manual and gaze input cascaded (MAGIC) pointing , 1999, CHI '99.

[31]  Pingmei Xu,et al.  TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking , 2015, ArXiv.

[32]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[33]  Yoichi Sato,et al.  Appearance-Based Gaze Estimation With Online Calibration From Mouse Operations , 2015, IEEE Transactions on Human-Machine Systems.

[34]  Mario Fritz,et al.  It’s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[35]  Oleg V. Komogortsev,et al.  Usability evaluation of eye tracking on an unmodified common tablet , 2013, CHI Extended Abstracts.

[36]  Peter D. Lawrence,et al.  A single camera eye-gaze tracking system with free head motion , 2006, ETRA.

[37]  Stephen Chi-fai Chan,et al.  Building a Personalized, Auto-Calibrating Eye Tracker from User Interactions , 2016, CHI.