On generalizing driver gaze zone estimation using convolutional neural networks

The knowledge of driver distraction will be important for self driving cars in the near future to determine the handoff time to the driver. Driver's gaze direction has been previously shown as an important cue in understanding distraction. While there has been a significant improvement in personalized driver gaze zone estimation systems, a generalized gaze zone estimation system which is invariant to different subjects, perspective and scale is still lagging behind. We take a step towards the generalized system using a Convolutional Neural Network (CNN). For evaluating our system, we collect large naturalistic driving data of 11 drives, driven by 10 subjects in two different cars and label gaze zones for 47515 frames. We train our CNN on 7 subjects and test on the other 3 subjects. Our best performing model achieves an accuracy of 93.36% showing good generalization capability.

[1]  Rebecca L Olson,et al.  The Impact of Hand-Held and Hands-Free Cell Phone Use on Driving Performance and Safety-Critical Event Risk , 2013 .

[2]  Mohan M. Trivedi,et al.  Gaze fixations and dynamics for behavior modeling and prediction of on-road driving maneuvers , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[3]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Miguel Ángel Sotelo,et al.  Real-time system for monitoring driver vigilance , 2004, Proceedings of the IEEE International Symposium on Industrial Electronics, 2005. ISIE 2005..

[5]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[6]  Qiang Ji,et al.  Real-Time Eye, Gaze, and Face Pose Tracking for Monitoring Driver Vigilance , 2002, Real Time Imaging.

[7]  Neville A. Stanton,et al.  Takeover Time in Highly Automated Vehicles: Noncritical Transitions to and From Manual Control , 2017, Hum. Factors.

[8]  Mohan M. Trivedi,et al.  Tactical driver behavior prediction and intent inference: A review , 2011, 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[9]  Kang Ryoung Park,et al.  Real-Time Gaze Estimator Based on Driver's Head Orientation for Forward Collision Warning System , 2011, IEEE Transactions on Intelligent Transportation Systems.

[10]  Mohan M. Trivedi,et al.  Where is the driver looking: Analysis of head, eye and iris for robust gaze zone estimation , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[11]  Albert Kircher,et al.  A Gaze-Based Driver Distraction Warning System and Its Effect on Visual Behavior , 2013, IEEE Transactions on Intelligent Transportation Systems.

[12]  Mubarak Shah,et al.  Determining driver visual attention with one camera , 2003, IEEE Trans. Intell. Transp. Syst..

[13]  Mohan M. Trivedi,et al.  Robust and continuous estimation of driver gaze zone by dynamic analysis of multiple face videos , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[14]  Carlos Busso,et al.  Detecting Drivers' Mirror-Checking Actions and Its Application to Maneuver and Secondary Task Recognition , 2016, IEEE Transactions on Intelligent Transportation Systems.

[15]  Keiichi Uchimura,et al.  Driver inattention monitoring system for intelligent vehicles: A review , 2009 .

[16]  Alex Fridman,et al.  Driver Gaze Estimation Without Using Eye Movement , 2015, ArXiv.

[17]  Neil K Chaudhary,et al.  Effect of Passenger Presence on Older Drivers’ Risk of Fatal Crash Involvement , 2014, Traffic injury prevention.

[18]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Qiang Ji,et al.  Real Time Visual Cues Extraction for Monitoring Driver Vigilance , 2001, ICVS.

[20]  Mohan M. Trivedi,et al.  On driver gaze estimation: Explorations and fusion of geometric and data driven approaches , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[21]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Mohan M. Trivedi,et al.  Looking at Humans in the Age of Self-Driving and Highly Automated Vehicles , 2016, IEEE Transactions on Intelligent Vehicles.

[23]  Takahiro Ishikawa,et al.  Passive driver gaze tracking with active appearance models , 2004 .

[24]  Sung Kyung Hong,et al.  Real-time categorization of driver's gaze zone using the deep learning techniques , 2016, 2016 International Conference on Big Data and Smart Computing (BigComp).

[25]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[26]  Pablo Lardelli-Claret,et al.  The influence of passengers on the risk of the driver causing a car collision in Spain. Analysis of collisions from 1990 to 1999. , 2004, Accident; analysis and prevention.

[27]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[28]  Mohan M. Trivedi,et al.  Looking at faces in a vehicle: A deep CNN based approach and evaluation , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[29]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[30]  Alex Fridman,et al.  'Owl' and 'Lizard': patterns of head pose and eye pose in driver gaze classification , 2015, IET Comput. Vis..

[31]  Carlos Hitoshi Morimoto,et al.  Pupil detection and tracking using multiple light sources , 2000, Image Vis. Comput..