Assessment of Driver Mental Fatigue Using Facial Landmarks

Driver fatigue is one of the causal factors for traffic accidents. Actions of facial units convey various information from our brains. This paper proposed a comprehensive approach to explore whether pattern of sequences of the driver’s facial landmarks changes from the alert start to the fatigue state. A driving-simulator-based experiment was designed and conducted. Videos of 21 participants’ faces were recorded during the experiment, together with subjective and others’ assessment of the level of alertness. Sequences of eye aspect ratio (EAR) and mouth aspect ratio (MAR) were calculated based on facial landmarks. Totally 21 feature candidates including Percent eye-closure over a fixed time window (PERCLOS), blink rate, statistics of blink duration, closing speed, reopening speed and number of yawns were extracted. A mental fatigue assessment model is proposed after feature selection. Four machine learning algorithms were used to build the assessment model of fatigue. The best performance was achieved by logistic regression, with cross-validation accuracies of 83.7% and 85.4%. This study may contribute to the development of driver fatigue monitoring system for automotive ergonomics.

[1]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  P. Caffier,et al.  Experimental evaluation of eye-blink parameters as a drowsiness measure , 2003, European Journal of Applied Physiology.

[3]  Xiongkuo Min,et al.  Eye Fatigue Assessment Using Unobtrusive Eye Tracker , 2018, IEEE Access.

[4]  K. Bengler,et al.  Vigilance Decrement and Passive Fatigue Caused by Monotony in Automated Driving , 2015 .

[5]  Ying Wu,et al.  Automatic Detection of Driver Fatigue Using Driving Operation Information for Transportation Safety , 2017, Sensors.

[6]  Carryl L. Baldwin,et al.  Driver fatigue: The importance of identifying causal factors of fatigue when considering detection and countermeasure technologies , 2009 .

[7]  D. Schroeder,et al.  Blink Rate: A Possible Measure of Fatigue , 1994, Human factors.

[8]  Xiaopei Wu,et al.  Driver Drowsiness Detection Using Multi-Channel Second Order Blind Identifications , 2019, IEEE Access.

[9]  Qiang Ji,et al.  A probabilistic framework for modeling and real-time monitoring human fatigue , 2006, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[10]  Jan Cech,et al.  Real-Time Eye Blink Detection using Facial Landmarks , 2016 .

[11]  Peter Rossiter,et al.  Applying neural network analysis on heart rate variability data to assess driver fatigue , 2011, Expert Syst. Appl..

[12]  Gamini Dissanayake,et al.  Driver Drowsiness Classification Using Fuzzy Wavelet-Packet-Based Feature-Extraction Algorithm , 2011, IEEE Transactions on Biomedical Engineering.

[13]  Amin Mirza Boroujerdian,et al.  Effect of road geometry on driver fatigue in monotonous environments: A simulator study , 2018, Transportation Research Part F: Traffic Psychology and Behaviour.

[14]  Weiqiang Zhang,et al.  Detection of mental fatigue state with wearable ECG devices , 2018, Int. J. Medical Informatics.

[15]  Azhar Quddus,et al.  Non-Intrusive Detection of Drowsy Driving Based on Eye Tracking Data , 2019, Transportation Research Record: Journal of the Transportation Research Board.

[16]  Frank K. Moss,et al.  The eyelid reflex as a criterion of ocular fatigue , 1937 .

[17]  R. Schleicher,et al.  Blinks and saccades as indicators of fatigue in sleepiness warners: looking tired? , 2022 .

[18]  Micheal Drieberg,et al.  A Hybrid Approach to Detect Driver Drowsiness Utilizing Physiological Signals to Improve System Performance and Wearability , 2017, Sensors.

[19]  Shiwu Li,et al.  Research on the Relationship between Reaction Ability and Mental State for Online Assessment of Driving Fatigue , 2016, International journal of environmental research and public health.

[20]  Ann Williamson,et al.  The link between fatigue and safety. , 2011, Accident; analysis and prevention.

[21]  Zuojin Li,et al.  Driver fatigue Detection using Approximate Entropic of steering wheel angle from Real driving Data , 2017, Int. J. Robotics Autom..

[22]  Simon G Hosking,et al.  Predicting driver drowsiness using vehicle measures: recent insights and future challenges. , 2009, Journal of safety research.

[23]  Jianfeng Hu,et al.  Real-time eye tracking for the assessment of driver fatigue , 2018, Healthcare technology letters.

[24]  Céline Craye,et al.  A Multi-Modal Driver Fatigue and Distraction Assessment System , 2015, International Journal of Intelligent Transportation Systems Research.

[25]  Anna Anund,et al.  Deriving heart rate variability indices from cardiac monitoring—An indicator of driver sleepiness , 2019, Traffic injury prevention.

[26]  Hikmat Ullah Khan,et al.  A Survey on State-of-the-Art Drowsiness Detection Techniques , 2019, IEEE Access.

[27]  Frans Coenen,et al.  Video-Based Classification of Driving Behavior Using a Hierarchical Classification System with Multiple Features , 2016, Int. J. Pattern Recognit. Artif. Intell..

[28]  Anwar M. Mirza,et al.  Fully automated real time fatigue detection of drivers through Fuzzy Expert Systems , 2014, Appl. Soft Comput..

[29]  Zuojin Li,et al.  Online Detection of Driver Fatigue Using Steering Wheel Angles for Real Driving Conditions , 2017, Sensors.

[30]  R J Fairbanks,et al.  RESEARCH ON VEHICLE-BASED DRIVER STATUS/PERFORMANCE MONITORING; DEVELOPMENT, VALIDATION, AND REFINEMENT OF ALGORITHMS FOR DETECTION OF DRIVER DROWSINESS. FINAL REPORT , 1994 .

[31]  Mali,et al.  Non-intrusive Detection and Prediction of Driver’s Fatigue Using Optimized Yawning Technique , 2017 .

[32]  W W Wierwille,et al.  Evaluation of driver drowsiness by trained raters. , 1994, Accident; analysis and prevention.

[33]  M. Chung,et al.  Electroencephalographic study of drowsiness in simulated driving with sleep deprivation , 2005 .

[34]  John D Lee,et al.  A contextual and temporal algorithm for driver drowsiness detection. , 2018, Accident; analysis and prevention.

[35]  David F. Dinges,et al.  Perclos: A valid psychophysiological measure of alertness as assessed by psychomotor vigilance , 1998 .

[36]  Carlos Fuentes-Silva,et al.  Dynamic set point model for driver alert state using digital image processing , 2019, Multimedia Tools and Applications.

[37]  Faramarz GHARAGOZLOU,et al.  Detecting Driver Mental Fatigue Based on EEG Alpha Power Changes during Simulated Driving , 2015, Iranian journal of public health.

[38]  Shigang Wang,et al.  Fatigue State Detection Based on Multi-Index Fusion and State Recognition Network , 2019, IEEE Access.

[39]  Lin Hong,et al.  Fatigue driving detection model based on multi‐feature fusion and semi‐supervised active learning , 2019, IET Intelligent Transport Systems.

[40]  Chengcheng Hua,et al.  Assessment of driver drowsiness using electroencephalogram signals based on multiple functional brain networks. , 2018, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[41]  Atsuo Murata,et al.  Proposal of a Method to Predict Subjective Rating on Drowsiness Using Physiological and Behavioral Measures , 2016 .

[42]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[43]  Jianfeng Hu,et al.  Automated detection of driver fatigue based on EEG signals using gradient boosting decision tree model , 2018, Cognitive Neurodynamics.

[44]  Michael G. Lenné,et al.  Predicting drowsiness-related driving events: a review of recent research methods and future opportunities , 2016 .