Unobtrusive and pervasive video-based eye-gaze tracking

Abstract Eye-gaze tracking has long been considered a desktop technology that finds its use inside the traditional office setting, where the operating conditions may be controlled. Nonetheless, recent advancements in mobile technology and a growing interest in capturing natural human behaviour have motivated an emerging interest in tracking eye movements within unconstrained real-life conditions, referred to as pervasive eye-gaze tracking. This critical review focuses on emerging passive and unobtrusive video-based eye-gaze tracking methods in recent literature, with the aim to identify different research avenues that are being followed in response to the challenges of pervasive eye-gaze tracking. Different eye-gaze tracking approaches are discussed in order to bring out their strengths and weaknesses, and to identify any limitations, within the context of pervasive eye-gaze tracking, that have yet to be considered by the computer vision community.

[1]  Ioannis Rigas,et al.  Current research in eye movement biometrics: An analysis based on BioEye 2015 competition , 2017, Image Vis. Comput..

[2]  Yanxia Zhang,et al.  Pupil-canthi-ratio: a calibration-free method for tracking horizontal gaze direction , 2014, AVI.

[3]  Kang Ryoung Park,et al.  A Novel Gaze Tracking Method Based on the Generation of Virtual Calibration Points , 2013, Sensors.

[4]  Takeshi Sasaki,et al.  Eye gaze estimation based on ellipse fitting and three-dimensional model of eye for “Intelligent Poster” , 2014, 2014 IEEE/ASME International Conference on Advanced Intelligent Mechatronics.

[5]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Krzysztof Fornalczyk,et al.  Single web camera robust interactive eye-gaze tracking method , 2015 .

[7]  Jian-Gang Wang,et al.  Estimating the eye gaze from one eye , 2005, Comput. Vis. Image Underst..

[8]  Aude Billard,et al.  Calibration-Free Eye Gaze Direction Detection with Gaussian Processes , 2008, VISAPP.

[9]  Mohammad Hassan Moradi,et al.  Eye Tracking Using A Novel Approach , 2007 .

[10]  Tiziana D'Orazio,et al.  An algorithm for real time eye detection in face images , 2004, ICPR 2004.

[11]  Kentaro Takemura,et al.  Estimating point-of-regard using corneal surface image , 2014, ETRA.

[12]  Jörg Müller,et al.  Eye tracking for public displays in the wild , 2015, Personal and Ubiquitous Computing.

[13]  Dietrich Paulus,et al.  Pupil center detection in low resolution images , 2010, ETRA.

[14]  Kang Ryoung Park,et al.  Real-Time Gaze Estimator Based on Driver's Head Orientation for Forward Collision Warning System , 2011, IEEE Transactions on Intelligent Transportation Systems.

[15]  R. S. Ramakrishna,et al.  Vision-based eye-gaze tracking for human computer interface , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[16]  Qiang Ji,et al.  A Probabilistic Approach to Online Eye Gaze Tracking Without Explicit Personal Calibration , 2015, IEEE Transactions on Image Processing.

[17]  Hafiz Adnan Habib,et al.  Eye-Gaze and Augmented Reality Framework for Driver Assistance , 2013 .

[18]  Atsushi Nakazawa,et al.  I See What You See: Point of Gaze Estimation from Corneal Images , 2013, 2013 2nd IAPR Asian Conference on Pattern Recognition.

[19]  Igor S. Pandzic,et al.  Eye pupil localization with an ensemble of randomized trees , 2014, Pattern Recognit..

[20]  J. Elison,et al.  Eye tracking young children with autism. , 2012, Journal of visualized experiments : JoVE.

[21]  Oleg V. Komogortsev,et al.  Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network , 2010, CHI Extended Abstracts.

[22]  Theo Gevers,et al.  Calibration-Free Gaze Estimation Using Human Gaze Patterns , 2013, 2013 IEEE International Conference on Computer Vision.

[23]  Mubarak Shah,et al.  Monitoring head/eye motion for driver alertness with one camera , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[24]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[25]  Stephen Chi-fai Chan,et al.  Building a Personalized, Auto-Calibrating Eye Tracker from User Interactions , 2016, CHI.

[26]  Narendra Ahuja,et al.  Appearance-based eye gaze estimation , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[27]  Nilamani Bhoi,et al.  Template Matching based Eye Detection in Facial Image , 2010 .

[28]  M. Mazo,et al.  System for assisted mobility using eye movements based on electrooculography , 2002, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[29]  Lance Gentry,et al.  Marketing and Eye-tracking Machines: Research Opportunities for an Affordable Tool , 2007 .

[30]  Oleg V. Komogortsev,et al.  Eye tracking on unmodified common tablets: challenges and solutions , 2012, ETRA '12.

[31]  Michele Nappi,et al.  Eye movement analysis for human authentication: a critical survey , 2016, Pattern Recognit. Lett..

[32]  Erhardt Barth,et al.  Accurate Eye Centre Localisation by Means of Gradients , 2011, VISAPP.

[33]  Olivier White,et al.  Computation of gaze orientation under unrestrained head movements , 2007, Journal of Neuroscience Methods.

[34]  Zicheng Liu,et al.  Eye gaze tracking using an RGBD camera: a comparison with a RGB solution , 2014, UbiComp Adjunct.

[35]  Miad Faezipour,et al.  Classifiers comparison for a new eye gaze direction classification system , 2014, IEEE Long Island Systems, Applications and Technology (LISAT) Conference 2014.

[36]  Sunu Wibirama,et al.  EYE-MOTION TRACKING USING MOTION GRADIENT , 2009 .

[37]  Rainer Stiefelhagen,et al.  Gaze tracking for multimodal human-computer interaction , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[38]  Kazuhiko Hamamoto,et al.  Real time eye tracking using initial centroid and gradient analysis technique , 2009, 2009 6th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology.

[39]  Wojciech Matusik,et al.  Eye Tracking for Everyone , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Isaak Kavasidis,et al.  Improving mobile device interaction by eye tracking analysis , 2012, 2012 Federated Conference on Computer Science and Information Systems (FedCSIS).

[41]  Jean-Marc Odobez,et al.  Gaze estimation from multimodal Kinect data , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[42]  Yanxia Zhang,et al.  Towards pervasive eye tracking using low-level image features , 2012, ETRA '12.

[43]  Timo Schneider,et al.  Manifold Alignment for Person Independent Appearance-Based Gaze Estimation , 2014, 2014 22nd International Conference on Pattern Recognition.

[44]  R. Remmel,et al.  An Inexpensive Eye Movement Monitor Using the Scleral Search Coil Technique , 1984, IEEE Transactions on Biomedical Engineering.

[45]  Vladimir Vezhnevets,et al.  Robust and Accurate Eye Contour Extraction , 2003 .

[46]  Jian-Gang Wang,et al.  Gaze determination via images of irises , 2001, Image Vis. Comput..

[47]  Aurobinda Routray,et al.  Real-time eye gaze direction classification using convolutional neural network , 2016, 2016 International Conference on Signal Processing and Communications (SPCOM).

[48]  Nicu Sebe,et al.  Visual Gaze Estimation by Joint Head and Eye Information , 2010, 2010 20th International Conference on Pattern Recognition.

[49]  Qiang Ji,et al.  In the Eye of the Beholder: A Survey of Models for Eyes and Gaze , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Yoichi Sato,et al.  Appearance-Based Gaze Estimation via Uncalibrated Gaze Pattern Recovery , 2017, IEEE Transactions on Image Processing.

[51]  Edwige Pissaloux,et al.  Gaze estimation using local features and non-linear regression , 2012, 2012 19th IEEE International Conference on Image Processing.

[52]  Qian Chen,et al.  Tracking Iris Contour with a 3D Eye-Model for Gaze Estimation , 2007, ACCV.

[53]  Karen M. Evans,et al.  Collecting and Analyzing Eye-Tracking Data in Outdoor Environments , 2012 .

[54]  Haibo Wang,et al.  Tracking Eye Gaze under Coordinated Head Rotations with an Ordinary Camera , 2009, ACCV.

[55]  Takahiro Ishikawa,et al.  Passive driver gaze tracking with active appearance models , 2004 .

[56]  Jeffrey F. Cohn,et al.  Person-Independent 3D Gaze Estimation Using Face Frontalization , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[57]  Kenneth P. Camilleri,et al.  Model-free non-rigid head pose tracking by joint shape and pose estimation , 2016, Machine Vision and Applications.

[58]  Yoichi Sato,et al.  Learning-by-Synthesis for Appearance-Based 3D Gaze Estimation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[59]  Takahiro Okabe,et al.  Adaptive Linear Regression for Appearance-Based Gaze Estimation , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Theo Gevers,et al.  Accurate eye center location and tracking using isophote curvature , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[61]  Ravi Kothari,et al.  Detection of eye locations in unconstrained visual images , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[62]  Z. Hammal,et al.  Eyes and eyebrows parametric models for automatic segmentation , 2004, 6th IEEE Southwest Symposium on Image Analysis and Interpretation, 2004..

[63]  Alexander Zelinsky,et al.  3-D facial pose and gaze point estimation using a robust real-time tracking paradigm , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[64]  Andrew T. Duchowski,et al.  PETMEI 2011: the 1st international workshop on pervasive eye tracking and mobile eye-based interaction , 2011, UbiComp '11.

[65]  Adam Strupczewski,et al.  High Accuracy Head Pose Tracking Survey , 2014, AMT.

[66]  Lars Kai Hansen,et al.  Heuristics for speeding up gaze estimation , 2005 .

[67]  Ba Linh Nguyen.  Eye Gaze Tracking , 2009, 2009 IEEE-RIVF International Conference on Computing and Communication Technologies.

[68]  Hua Cai,et al.  An integrated head pose and eye gaze tracking approach to non-intrusive visual attention measurement for wide FOV simulators , 2010, Virtual Reality.

[69]  Kenneth P. Camilleri,et al.  On-screen point-of-regard estimation under natural head movement for a computer with integrated webcam , 2015 .

[70]  Kai Kunze,et al.  My reading life: towards utilizing eyetracking on unmodified tablets and phones , 2013, UbiComp.

[71]  Yue Gao,et al.  Estimating 3D Gaze Directions Using Unlabeled Eye Images via Synthetic Iris Appearance Fitting , 2016, IEEE Transactions on Multimedia.

[72]  Norihiro Hagita,et al.  Gaze tracking in wide area using multiple camera observations , 2012, ETRA '12.

[73]  Petros Maragos,et al.  Estimation of eye gaze direction angles based on active appearance models , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[74]  Sung-Jea Ko,et al.  Eyeball model-based iris center localization for visible image-based eye-gaze tracking systems , 2013, IEEE Transactions on Consumer Electronics.

[75]  Kenneth P. Camilleri,et al.  Cursor control by point-of-regard estimation for a computer with integrated webcam , 2014 .

[76]  Alexander Zelinsky,et al.  An algorithm for real-time stereo vision implementation of head pose and gaze direction measurement , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[77]  Qiang Ji,et al.  Probabilistic gaze estimation without active personal calibration , 2011, CVPR 2011.

[78]  Jin Kim,et al.  A 2D eye gaze estimation system with low-resolution webcam images , 2011, EURASIP J. Adv. Signal Process..

[79]  Qiang Ji,et al.  3D gaze estimation with a single camera without IR illumination , 2008, 2008 19th International Conference on Pattern Recognition.

[80]  Thai-Hoang Huynh.  A new eye gaze detection algorithm using PCA features and recurrent neural networks , 2013, 2013 IEEE Symposium on Computational Intelligence in Control and Automation (CICA).

[81]  Mike Adams,et al.  Disabled Students in Education: Technology, Transition, and Inclusivity , 2011 .

[82]  Nicu Sebe,et al.  Webcam-Based Visual Gaze Estimation , 2009, ICIAP.

[83]  Michele Nappi,et al.  A New Gaze Analysis Based Soft-Biometric , 2013, MCPR.

[84]  Hirotake Yamazoe,et al.  Remote gaze estimation with a single camera based on facial-feature tracking without special calibration actions , 2008, ETRA.

[85]  Peter Robinson,et al.  A 3D Morphable Eye Region Model for Gaze Estimation , 2016, ECCV.

[86]  Richard P. Wildes,et al.  Reliable and fast eye finding in close-up images , 2002, Object recognition supported by user interaction for service robots.

[87]  Yi-Ping Hung,et al.  Appearance-Based Gaze Tracking with Free Head Movement , 2014, 2014 22nd International Conference on Pattern Recognition.

[88]  Hans-Werner Gellersen,et al.  Pursuit calibration: making gaze calibration less tedious and more flexible , 2013, UIST.

[89]  Lijun Yin,et al.  Viewing direction estimation based on 3D eyeball construction for HRI , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[90]  Yu-Tzu Lin,et al.  Real-time eye-gaze estimation using a low-resolution webcam , 2012, Multimedia Tools and Applications.

[91]  Takahiro Okabe,et al.  A Head Pose-free Approach for Appearance-based Gaze Estimation , 2011, BMVC.

[92]  Anura P. Jayasumana,et al.  A neural network based real-time gaze tracker , 2002, J. Netw. Comput. Appl..

[93]  Moshe Eizenman,et al.  An Automatic Personal Calibration Procedure for Advanced Gaze Estimation Systems , 2010, IEEE Transactions on Biomedical Engineering.

[94]  Ba Linh Nguyen,et al.  Eye gaze tracking with free head movements using a single camera , 2010, SoICT '10.

[95]  Thai-Hoang Huynh,et al.  Eye-gaze detection with a single WebCAM based on geometry features extraction , 2010, 2010 11th International Conference on Control Automation Robotics & Vision.

[96]  S T Moore,et al.  Robust pupil center detection using a curvature algorithm. , 1999, Computer methods and programs in biomedicine.

[97]  Minho Lee,et al.  Gaze tracking based on pupil estimation using multilayer perception , 2011, The 2011 International Joint Conference on Neural Networks.

[98]  Jean-Marc Odobez,et al.  Geometric Generative Gaze Estimation (G3E) for Remote RGB-D Cameras , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[99]  Stefan Kohlbecher,et al.  Studying Gaze-based Human Robot Interaction: An Experimental Platform , 2012, HRI 2012.

[100]  Levente Hajder,et al.  Improving Human-Computer Interaction by gaze tracking , 2012, 2012 IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom).

[101]  Ioannis Patras,et al.  Gaze Tracking by Using Factorized Likelihoods Particle Filtering and Stereo Vision , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[102]  Bernhard Egger,et al.  Pose Normalization for Eye Gaze Estimation and Facial Attribute Description from Still Images , 2014, GCPR.

[103]  D. Paulus,et al.  Gaze-estimation for consumer-grade cameras using a Gaussian process latent variable model , 2016, Pattern Recognition and Image Analysis.

[104]  Nicu Sebe,et al.  What Are You Looking at? , 2011, International Journal of Computer Vision.

[105]  Shao-Yi Chien,et al.  Real-time eye localization, blink detection, and gaze estimation system without infrared illumination , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[106]  Michèle Molina,et al.  Appearance-based gaze tracking with spectral clustering and semi-supervised Gaussian process regression , 2013, ETSA '13.

[107]  Kenneth P. Camilleri,et al.  Model-based head pose-free gaze estimation for assistive communication , 2016, Comput. Vis. Image Underst..

[108]  Xiaowu Chen,et al.  Person-independent eye gaze prediction from eye images using patch-based features , 2016, Neurocomputing.

[109]  Yanxia Zhang,et al.  SideWays: a gaze interface for spontaneous interaction with situated displays , 2013, CHI.

[110]  Takahiro Okabe,et al.  Inferring human gaze from appearance via adaptive linear regression , 2011, 2011 International Conference on Computer Vision.

[111]  Yoichi Sato,et al.  Appearance-Based Gaze Estimation With Online Calibration From Mouse Operations , 2015, IEEE Transactions on Human-Machine Systems.

[112]  Heiko Neumann,et al.  Detection of Head Pose and Gaze Direction for Human-Computer Interaction , 2006, PIT.

[113]  Jasbir Kaur,et al.  Eye tracking based driver fatigue monitoring and warning system , 2011, India International Conference on Power Electronics 2010 (IICPE2010).

[114]  Frederick C. Harris,et al.  Eye Detection using Wavelets and ANN , 2004 .

[115]  Antonio Albiol,et al.  Estimating Point of Regard with a Consumer Camera at a Distance , 2013, IbPRIA.

[116]  Peter Corcoran,et al.  Towards the development of a standardized performance evaluation framework for eye gaze estimation systems in consumer platforms , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[117]  Yi Liu,et al.  A new reconstruction method in gaze estimation with natural head movement , 2017, 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA).

[118]  Peter Husar,et al.  Real-time calibration-free autonomous eye tracker , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[119]  Jie Zhu,et al.  Subpixel eye gaze tracking , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[120]  Randa Atta,et al.  An efficient eye detection and tracking system based on particle swarm optimization and adaptive block-matching search algorithm , 2014, Eng. Appl. Artif. Intell..

[121]  Florian Alt,et al.  EyeScout: Active Eye Tracking for Position and Movement Independent Gaze Interaction with Large Public Displays , 2017, UIST.

[122]  James Hays,et al.  WebGazer: Scalable Webcam Eye Tracking Using User Interactions , 2016, IJCAI.

[123]  Z Tomori,et al.  Eye tracking using artificial neural networks for human computer interaction. , 2011, Physiological research.

[124]  Lei Huang,et al.  Gaze Estimation Based on 3D Face Structure and Pupil Centers , 2014, 2014 22nd International Conference on Pattern Recognition.

[125]  Atif Bin Mansoor,et al.  Real Time Eyes Tracking and Classification for Driver Fatigue Detection , 2008, ICIAR.

[126]  Peter Robinson,et al.  Rendering of Eyes for Eye-Shape Registration and Gaze Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[127]  Yoichi Sato,et al.  Calibration-free gaze sensing using saliency maps , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[128]  Kostas Karpouzis,et al.  Visual Focus of Attention in Non-calibrated Environments using Gaze Estimation , 2014, International Journal of Computer Vision.

[129]  Qiong Huang,et al.  TabletGaze: Unconstrained Appearance-based Gaze Estimation in Mobile Tablets , 2015 .

[130]  Nicu Sebe,et al.  Combining Head Pose and Eye Location Information for Gaze Estimation , 2012, IEEE Transactions on Image Processing.

[131]  David R. Flatla,et al.  Calibration games: making calibration tasks enjoyable by adding motivating game elements , 2011, UIST.

[132]  Robert J. K. Jacob,et al.  What you look at is what you get: eye movement-based interaction techniques , 1990, CHI '90.

[133]  Stefanos Kollias,et al.  A natural head pose and eye gaze dataset , 2009, AFFINE '09.

[134]  Takahiko Horiuchi,et al.  Eye-Gaze Detection from Monocular Camera Image Using Parametric Template Matching , 2007, ACCV.

[135]  Klaus D. Tönnies,et al.  Feasibility of Hough-transform-based iris localisation for real-time-application , 2002, Object recognition supported by user interaction for service robots.

[136]  Li-Qun Xu,et al.  A Novel Approach to Real-time Non-intrusive Gaze Finding , 1998, BMVC.

[137]  Stephen Chi-fai Chan,et al.  Building a Self-Learning Eye Gaze Model from User Interaction Data , 2014, ACM Multimedia.

[138]  Ioannis Rigas,et al.  Towards a multi-source fusion approach for eye movement-driven recognition , 2016, Inf. Fusion.

[139]  Peter Robinson,et al.  Learning an appearance-based gaze estimator from one million synthesised images , 2016, ETRA.

[140]  Zakia Hammal,et al.  Eyes Segmentation Applied to Gaze Direction and Vigilance Estimation , 2005, ICAPR.

[141]  Stefan Kohlbecher,et al.  Calibration-free eye tracking by reconstruction of the pupil ellipse in 3D space , 2008, ETRA.

[142]  Serge Miguet,et al.  Eye and Gaze Tracking Algorithm for Collaborative Learning System , 2006 .

[143]  Eric M. Schwartz,et al.  Eye Detection Using Morphological and Color Image Processing , 2009 .

[144]  Shaun J. Canavan,et al.  A Multi-Gesture Interaction System Using a 3-D Iris Disk Model for Gaze Estimation and an Active Appearance Model for 3-D Hand Pointing , 2011, IEEE Transactions on Multimedia.

[145]  Jing Xiao,et al.  Meticulously detailed eye region model and its application to analysis of facial images , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[146]  Jean-Marc Odobez,et al.  3D Gaze Tracking and Automatic Gaze Coding from RGB-D Cameras , 2014, CVPR 2014.

[147]  Jean-Marc Odobez,et al.  Person independent 3D gaze estimation from remote RGB-D cameras , 2013, 2013 IEEE International Conference on Image Processing.

[148]  Andreas Bulling,et al.  EyeTab: model-based gaze estimation on unmodified tablet computers , 2014, ETRA.