Driver Drowsiness Detection Using Condition-Adaptive Representation Learning Framework

We propose a condition-adaptive representation learning framework for driver drowsiness detection based on a 3D-deep convolutional neural network. The proposed framework consists of four models: spatio-temporal representation learning, scene condition understanding, feature fusion, and drowsiness detection. Spatio-temporal representation learning extracts features that can describe motions and appearances in video simultaneously. Scene condition understanding classifies the scene conditions related to various conditions about the drivers and driving situations, such as statuses of wearing glasses, illumination condition of driving, and motion of facial elements, such as head, eye, and mouth. Feature fusion generates a condition-adaptive representation using two features extracted from the above models. The drowsiness detection model recognizes driver drowsiness status using the condition-adaptive representation. The condition-adaptive representation learning framework can extract more discriminative features focusing on each scene condition than the general representation so that the drowsiness detection method can provide more accurate results for the various driving situations. The proposed framework is evaluated with the NTHU drowsy driver detection video dataset. The experimental results show that our framework outperforms the existing drowsiness detection methods based on visual analysis.

[1]  Shin Yamamoto,et al.  Measurement of Driver's Consciousness by Image Processing -A Method for Presuming Driver's Drowsiness by Eye-Blinks coping with Individual Differences - , 2006, 2006 IEEE International Conference on Systems, Man and Cybernetics.

[2]  Francesco Lamonaca,et al.  Detection of the eye blinks for human's fatigue monitoring , 2012, 2012 IEEE International Symposium on Medical Measurements and Applications Proceedings.

[3]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  John D Lee,et al.  A contextual and temporal algorithm for driver drowsiness detection. , 2018, Accident; analysis and prevention.

[5]  Ayoub Al-Hamadi,et al.  A color based approach for eye blink detection in image sequences , 2011, 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA).

[6]  Seunghoon Hong,et al.  Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[8]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[9]  Chrysoula Kourtidou-Papadeli,et al.  Monitoring sleepiness with on-board electrophysiological recordings for preventing sleep-deprived traffic accidents , 2007, Clinical Neurophysiology.

[10]  Rainer Lienhart,et al.  An extended set of Haar-like features for rapid object detection , 2002, Proceedings. International Conference on Image Processing.

[11]  Andrew M. Pack,et al.  Characteristics of crashes attributed to the driver having fallen asleep. , 1995, Accident; analysis and prevention.

[12]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[13]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[14]  Omer Tsimhoni,et al.  Model-Based Analysis and Classification of Driver Distraction Under Secondary Tasks , 2010, IEEE Transactions on Intelligent Transportation Systems.

[15]  David F. Dinges,et al.  Perclos: A valid psychophysiological measure of alertness as assessed by psychomotor vigilance , 1998 .

[16]  Peter Rossiter,et al.  Applying neural network analysis on heart rate variability data to assess driver fatigue , 2011, Expert Syst. Appl..

[17]  Matti Pietikäinen,et al.  Dynamic texture and scene classification by transferring deep image features , 2015, Neurocomputing.

[18]  Jian Sun,et al.  Convolutional neural networks at constrained time cost , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Lorenzo Torresani,et al.  Learning Spatiotemporal Features with 3D Convolutional Networks , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[20]  F Sagberg,et al.  Road accidents caused by drivers falling asleep. , 1999, Accident; analysis and prevention.

[21]  Dumitru Erhan,et al.  Scalable Object Detection Using Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Kazuya Takeda,et al.  Driver identification using driving behavior signals , 2005, Proceedings. 2005 IEEE Intelligent Transportation Systems, 2005..

[23]  Simon G Hosking,et al.  Predicting driver drowsiness using vehicle measures: recent insights and future challenges. , 2009, Journal of safety research.

[24]  Chong-Wah Ngo,et al.  Human Action Recognition in Unconstrained Videos by Explicit Motion Modeling , 2015, IEEE Transactions on Image Processing.

[25]  Seong G. Kong,et al.  Visual Analysis of Eye State and Head Pose for Driver Alertness Monitoring , 2013, IEEE Transactions on Intelligent Transportation Systems.

[26]  Li-Jia Li,et al.  Multi-view Face Detection Using Deep Convolutional Neural Networks , 2015, ICMR.

[27]  Ping Wang,et al.  A method of detecting driver drowsiness state based on multi-features of face , 2012, 2012 5th International Congress on Image and Signal Processing.

[28]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[29]  Evgeny Burnaev,et al.  Large-Scale Shape Retrieval with Sparse 3D Convolutional Neural Networks , 2017, AIST.

[30]  Jarek Krajewski,et al.  Detecting Sleepy Drivers by Pattern Recognition based Analysis of Steering Wheel Behaviour , 2010 .

[31]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Gamini Dissanayake,et al.  Driver Drowsiness Classification Using Fuzzy Wavelet-Packet-Based Feature-Extraction Algorithm , 2011, IEEE Transactions on Biomedical Engineering.

[33]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Nicu Sebe,et al.  Learning Deep Representations of Appearance and Motion for Anomalous Event Detection , 2015, BMVC.

[37]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[38]  Sangwook Lee,et al.  Representation Learning, Scene Understanding, and Feature Fusion for Drowsiness Detection , 2016, ACCV Workshops.

[39]  Yoshimi Furukawa,et al.  Estimate of driver's fatigue through steering motion , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[40]  Fei Pan,et al.  Driver Drowsiness Detection System Based on Feature Representation Learning Using Various Deep Networks , 2016, ACCV Workshops.

[41]  Behnoosh Hariri,et al.  YawDD: a yawning detection dataset , 2014, MMSys '14.

[42]  Yvonne Tran,et al.  Improving classification rates for use in fatigue countermeasure devices using brain activity , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[43]  Patrick Bouthemy,et al.  Better Exploiting Motion for Better Action Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Azim Eskandarian,et al.  Unobtrusive drowsiness detection by neural network learning of driver steering , 2001 .

[45]  G Maycock,et al.  Sleepiness and driving: the experience of UK car drivers , 1996, Journal of sleep research.

[46]  Xiaoou Tang,et al.  Learning Deep Representation for Face Alignment with Auxiliary Attributes , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Yong Du,et al.  Hierarchical recurrent neural network for skeleton based action recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Luis Miguel Bergasa,et al.  Vision-based drowsiness detector for real driving conditions , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[49]  Amit Sethi,et al.  Drowsy driver detection using representation learning , 2014, 2014 IEEE International Advance Computing Conference (IACC).

[50]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.

[51]  Lidia P. Kostyniuk,et al.  National Survey on Distracted Driving Attitudes and Behaviors -- 2012 , 2013 .

[52]  Stefanos Zafeiriou,et al.  A comparison of different features for automatic eye blinking detection with an application to analysis of deceptive behavior , 2012, 2012 5th International Symposium on Communications, Control and Signal Processing.

[53]  Zhi-Hong Mao,et al.  Detection of Driver Fatigue Caused by Sleep Deprivation , 2009, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[54]  Jitendra Malik,et al.  Region-Based Convolutional Networks for Accurate Object Detection and Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Cordelia Schmid,et al.  Action Recognition with Improved Trajectories , 2013, 2013 IEEE International Conference on Computer Vision.

[56]  Pavlo Molchanov,et al.  Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).