Deep Fusion for 3D Gaze Estimation From Natural Face Images Using Multi-Stream CNNs

Over the last few decades, eye gaze estimation techniques have been thoroughly investigated by many researchers. However, predicting a 3D gaze from a 2D natural image remains challenging because it has to deal with several issues such as diverse head positions, face shape transformation, illumination variations, and subject individuality. Many previous studies employ convolutional neural networks (CNNs) for this task, and yet the accuracy needs improvement for its practical use. In this paper, we propose a 3D gaze estimation framework based on the data science perspective: First, a novel neural network architecture is designed to exploit every possible visual attribute such as the states of both eyes and the head position, including several augmentations; secondly, the data fusion method is utilized by incorporating multiple gaze datasets. Extensive experiments were carried out using two standard eye gaze datasets, including comparative analysis. The experimental results suggest that our method outperforms state-of-the-art with 2.8 degrees for MPIIGaze and 3.05 degrees for EYEDIAP dataset, respectively, indicating that it has a potential for real applications.