A Dual-Cameras-Based Driver Gaze Mapping System With an Application on Non-Driving Activities Monitoring

Characterisation of the driver’s non-driving activities (NDAs) is of great importance to the design of the take-over control strategy in Level 3 automation. Gaze estimation is a typical approach to monitor the driver’s behaviour since the eye gaze is normally engaged with the human activities. However, current eye gaze tracking techniques are either costly or intrusive which limits their applicability in vehicles. This paper proposes a low-cost and non-intrusive dual-cameras based gaze mapping system that visualises the driver’s gaze using a heat map. The challenges introduced by complex head movement during NDAs and camera distortion are addressed by proposing a nonlinear polynomial model to establish the relationship between the face features and eye gaze on the simulated driver’s view. The Root Mean Square Error of this system in the in-vehicle experiment for the X and Y direction is 7.80±5.99 pixel and 4.64±3.47 pixel respectively with the image resolution of $1440 \times 1080$ pixels. This system is successfully demonstrated to evaluate three NDAs with visual attention. This technique, acting as a generic tool to monitor driver’s visual attention, will have wide applications on NDA characterisation for intelligent design of take over strategy and driving environment awareness for current and future automated vehicles.

[1]  Peter Robinson,et al.  Rendering of Eyes for Eye-Shape Registration and Gaze Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Nikos Fakotakis,et al.  Precise localization of eye centers in low resolution color images , 2015, Image Vis. Comput..

[3]  Hongbin Zha,et al.  Specialized gaze estimation for children by convolutional neural network and domain adaptation , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[4]  Min Young Kim,et al.  Selective attentional point-tracking through a head-mounted stereo gaze tracker based on trinocular epipolar geometry , 2015, 2015 IEEE International Instrumentation and Measurement Technology Conference (I2MTC) Proceedings.

[5]  Degui Xiao,et al.  Detection of drivers visual attention using smartphone , 2016, 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD).

[6]  Fernando De la Torre,et al.  Driver Gaze Tracking and Eyes Off the Road Detection System , 2015, IEEE Transactions on Intelligent Transportation Systems.

[7]  Peter Corcoran,et al.  A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms , 2017, IEEE Access.

[8]  Lee Skrypchuk,et al.  An Orientation Sensor-Based Head Tracking System for Driver Behaviour Monitoring , 2017, Sensors.

[9]  Jing Li,et al.  Appearance-based gaze block estimation via CNN classification , 2017, 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP).

[10]  Bryan Reimer,et al.  A Framework for Robust Driver Gaze Classification , 2016 .

[11]  Kang Ryoung Park,et al.  Vision-based method for detecting driver drowsiness and distraction in driver monitoring system , 2011 .

[12]  Peter Robinson,et al.  Constrained Local Neural Fields for Robust Facial Landmark Detection in the Wild , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[13]  Mario Fritz,et al.  Appearance-based gaze estimation in the wild , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Marco La Cascia,et al.  Fast, Reliable Head Tracking under Varying Illumination: An Approach Based on Registration of Texture-Mapped 3D Models , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Takahiro Ishikawa,et al.  Detecting driver's visual attention area by using vehicle-mounted device , 2017, 2017 IEEE 16th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC).

[16]  Qijie Zhao,et al.  A Novel Approach to Estimate Gaze Direction in Eye Gaze HCI System , 2013, 2013 5th International Conference on Intelligent Human-Machine Systems and Cybernetics.

[17]  Louis-Philippe Morency,et al.  OpenFace 2.0: Facial Behavior Analysis Toolkit , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[18]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Albert Kircher,et al.  A Gaze-Based Driver Distraction Warning System and Its Effect on Visual Behavior , 2013, IEEE Transactions on Intelligent Transportation Systems.

[20]  Agata Marta Soccini.  Gaze estimation based on head movements in virtual reality applications using deep learning , 2017, 2017 IEEE Virtual Reality (VR).

[21]  Mohan M. Trivedi,et al.  On the Roles of Eye Gaze and Head Dynamics in Predicting Driver's Intent to Change Lanes , 2009, IEEE Transactions on Intelligent Transportation Systems.

[22]  Noelia Hernández,et al.  Gaze Fixation System for the Evaluation of Driver Distractions Induced by IVIS , 2012, IEEE Transactions on Intelligent Transportation Systems.

[23]  Qiang Ji,et al.  In the Eye of the Beholder: A Survey of Models for Eyes and Gaze , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Daniel Weiskopf,et al.  Hilbert attention maps for visualizing spatiotemporal gaze data , 2016, 2016 IEEE Second Workshop on Eye Tracking and Visualization (ETVIS).

[25]  S.M. Kim,et al.  Non-intrusive eye gaze tracking under natural head movements , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[26]  Hanaa Mohsin,et al.  Pupil detection algorithm based on feature extraction for eye gaze , 2017, 2017 6th International Conference on Information and Communication Technology and Accessibility (ICTA).

[27]  Michael Burch,et al.  Visual Analytics Methodology for Eye Movement Studies , 2012, IEEE Transactions on Visualization and Computer Graphics.

[28]  Peter D. Lawrence,et al.  A single camera eye-gaze tracking system with free head motion , 2006, ETRA.

[29]  Kang Ryoung Park,et al.  Deep Learning-Based Gaze Detection System for Automobile Drivers Using a NIR Camera Sensor , 2018, Sensors.

[30]  Peter Robinson,et al.  OpenFace: An open source facial behavior analysis toolkit , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[31]  Xiang Yang,et al.  Research on Pupil Center Localization in Eye Gaze Tracking System , 2018, 2018 37th Chinese Control Conference (CCC).

[32]  Mohan M. Trivedi,et al.  Driver Gaze Zone Estimation Using Convolutional Neural Networks: A General Framework and Ablative Analysis , 2018, IEEE Transactions on Intelligent Vehicles.

[33]  Alex Fridman,et al.  Driver Gaze Region Estimation without Use of Eye Movement , 2015, IEEE Intelligent Systems.

[34]  Andrew T Duchowski,et al.  A breadth-first survey of eye-tracking applications , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[35]  Tomas McKelvey,et al.  Driver-Gaze Zone Estimation Using Bayesian Filtering and Gaussian Processes , 2016, IEEE Transactions on Intelligent Transportation Systems.

[36]  Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles , 2022 .

[37]  M. Sivak,et al.  Motion Sickness in Self-Driving Vehicles , 2015 .