RITnet: Real-time Semantic Segmentation of the Eye for Gaze Tracking

Accurate eye segmentation can improve eye-gaze estimation and support interactive computing based on visual attention; however, existing eye segmentation methods suffer from issues such as person-dependent accuracy, lack of robustness, and an inability to be run in real-time. Here, we present the RITnet model, which is a deep neural network that combines U-Net and DenseNet. RITnet is under 1 MB and achieves 95.3% accuracy on the 2019 OpenEDS Semantic Segmentation challenge. Using a GeForce GTX 1080 Ti, RITnet tracks at > 300Hz, enabling real-time gaze tracking applications. Pre-trained models and source code are available

[1]  Karel J. Zuiderveld,et al.  Contrast Limited Adaptive Histogram Equalization , 1994, Graphics Gems.

[2]  Andreas Bulling,et al.  Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction , 2014, UbiComp Adjunct.

[3]  Andreas Bulling,et al.  EyeTab: model-based gaze estimation on unmodified tablet computers , 2014, ETRA.

[4]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[5]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[6]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[7]  Yoshua Bengio,et al.  The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  Sébastien Ourselin,et al.  Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations , 2017, DLMIA/ML-CDS@MICCAI.

[9]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Sylvain Paris,et al.  Deep Photo Style Transfer , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Otmar Hilliges,et al.  Deep Pictorial Gaze Estimation , 2018, ECCV.

[12]  Anjul Patney,et al.  Towards virtual reality infinite walking , 2018, ACM Trans. Graph..

[13]  Gregory Hughes,et al.  OpenEDS: Open Eye Dataset , 2019, ArXiv.

[14]  Zhengyang Wu,et al.  EyeNet: A Multi-Task Network for Off-Axis Eye Gaze Estimation and User Understanding , 2019, ArXiv.

[15]  Jan Kautz,et al.  Few-Shot Adaptive Gaze Estimation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  Jose Dolz,et al.  Boundary loss for highly unbalanced segmentation , 2018, MIDL.

[17]  Wolfgang Rosenstiel,et al.  500, 000 Images Closer to Eyelid and Pupil Segmentation , 2019, CAIP.

[18]  Joohwan Kim,et al.  NVGaze: An Anatomically-Informed Dataset for Low-Latency, Near-Eye Gaze Estimation , 2019, CHI.

[19]  J. Pelz,et al.  Motion tracking of iris features to detect small eye movements , 2019, Journal of eye movement research.

[20]  Jeff B. Pelz,et al.  Privacy-Preserving Eye Videos using Rubber Sheet Model , 2020, ETRA Short Papers.