Driver Gaze Estimation in the Real World: Overcoming the Eyeglass Challenge

A driver's gaze is critical for determining the driver's attention level, state, situational awareness, and readiness to take over control from partially and fully automated vehicles. Tracking both the head and eyes (pupils) can provide a reliable estimation of a driver's gaze using face images under ideal conditions. However, the vehicular environment introduces a variety of challenges that are usually unaccounted for - harsh illumination, nighttime conditions, and reflective/dark eyeglasses. Unfortunately, relying on head pose alone under such conditions can prove to be unreliable owing to significant eye movements. In this study, we offer solutions to address these problems encountered in the real world. To solve issues with lighting, we demonstrate that using an infrared camera with suitable equalization and normalization usually suffices. To handle eyeglasses and their corresponding artifacts, we adopt the idea of image-to-image translation using generative adversarial networks (GANs) to pre-process images prior to gaze estimation. To this end, we propose the Gaze Preserving CycleGAN (GPCycleGAN). This network preserves the driver's gaze while removing potential eyeglasses from infrared face images. Our approach exhibits improved performance and robustness on challenging real-world data spanning 13 subjects and a variety of driving conditions.

[1]  Mohan M. Trivedi,et al.  Looking-in and looking-out vision for Urban Intelligent Assistance: Estimation of driver attentive state and dynamic surround for safe merging and braking , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[2]  Nasser Kehtarnavaz,et al.  Low-Light Auto-Focus Enhancement for Digital and Cell-Phone Camera Image Pipelines , 2007, IEEE Transactions on Consumer Electronics.

[3]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[4]  Mohan M. Trivedi,et al.  Driver Gaze Zone Estimation Using Convolutional Neural Networks: A General Framework and Ablative Analysis , 2018, IEEE Transactions on Intelligent Vehicles.

[5]  Alex Fridman,et al.  Driver Gaze Region Estimation without Use of Eye Movement , 2015, IEEE Intelligent Systems.

[6]  Mingwu Ren,et al.  Unsupervised Eyeglasses Removal in the Wild , 2020, IEEE transactions on cybernetics.

[7]  Takeshi Saitoh,et al.  CNN-Based Pupil Center Detection for Wearable Gaze Estimation System , 2017, Appl. Comput. Intell. Soft Comput..

[8]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Aqing Yang,et al.  Deep Convolution Neural Networks for Automatic Eyeglasses Removal , 2017 .

[10]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[11]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Stan Z. Li,et al.  Learning sparse feature for eyeglasses problem in face recognition , 2011, Face and Gesture 2011.

[13]  Mohan Manubhai Trivedi,et al.  Dynamics of Driver's Gaze: Explorations in Behavior Modeling and Maneuver Prediction , 2018, IEEE Transactions on Intelligent Vehicles.

[14]  Peter Corcoran,et al.  A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms , 2017, IEEE Access.

[15]  Xianping Fu,et al.  Continuous Driver’s Gaze Zone Estimation Using RGB-D Camera , 2019, Sensors.

[16]  Mohan M. Trivedi,et al.  Robust and continuous estimation of driver gaze zone by dynamic analysis of multiple face videos , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[17]  Wei Shen,et al.  Learning Residual Images for Face Attribute Manipulation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Wai Keung Wong,et al.  Eyeglasses removal of thermal image based on visible information , 2013, Inf. Fusion.

[20]  Ling Liu,et al.  Effective Facial Obstructions Removal with Enhanced Cycle-Consistent Generative Adversarial Networks , 2018, AIMS.

[21]  David Zhang,et al.  Deep Identity-aware Transfer of Facial Attributes , 2016, ArXiv.

[22]  Smita Krishnaswamy,et al.  TraVeLGAN: Image-To-Image Translation by Transformation Vector Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Yoshua Bengio,et al.  Generative Adversarial Networks , 2014, ArXiv.

[24]  Keiichi Uchimura,et al.  Driver Inattention Monitoring System for Intelligent Vehicles: A Review , 2009, IEEE Transactions on Intelligent Transportation Systems.

[25]  Michael Fitzharris,et al.  Driver inattention and driver distraction in serious casualty crashes: data from the Australian National Crash In-depth Study. , 2013, Accident; analysis and prevention.

[26]  Carlos Busso,et al.  Probabilistic Estimation of the Gaze Region of the Driver using Dense Classification , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[27]  Takeo Kanade,et al.  Illumination-free gaze estimation method for first-person vision wearable device , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[28]  Kang Ryoung Park,et al.  Driver Gaze Detection Based on Deep Residual Networks Using the Combined Single Image of Dual Near-Infrared Cameras , 2019, IEEE Access.

[29]  Lior Wolf,et al.  Unsupervised Cross-Domain Image Generation , 2016, ICLR.

[30]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Kang Ryoung Park,et al.  Real-Time Gaze Estimator Based on Driver's Head Orientation for Forward Collision Warning System , 2011, IEEE Transactions on Intelligent Transportation Systems.

[32]  Mohan M. Trivedi,et al.  Where is the driver looking: Analysis of head, eye and iris for robust gaze zone estimation , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[33]  Mohan M. Trivedi,et al.  Head, Eye, and Hand Patterns for Driver Activity Recognition , 2014, 2014 22nd International Conference on Pattern Recognition.

[34]  Mohan M. Trivedi,et al.  Exploring the Situational Awareness of Humans inside Autonomous Vehicles , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[35]  Kang Ryoung Park,et al.  Deep Learning-Based Gaze Detection System for Automobile Drivers Using a NIR Camera Sensor , 2018, Sensors.

[36]  M. Amaç Güvensan,et al.  Driver Behavior Analysis for Safe Driving: A Survey , 2015, IEEE Transactions on Intelligent Transportation Systems.