Real-time nonintrusive monitoring and prediction of driver fatigue

This paper describes a real-time online prototype driver-fatigue monitor. It uses remotely located charge-coupled-device cameras equipped with active infrared illuminators to acquire video images of the driver. Various visual cues that typically characterize the level of alertness of a person are extracted in real time and systematically combined to infer the fatigue level of the driver. The visual cues employed characterize eyelid movement, gaze movement, head movement, and facial expression. A probabilistic model is developed to model human fatigue and to predict fatigue based on the visual cues obtained. The simultaneous use of multiple visual cues and their systematic combination yields a much more robust and accurate fatigue characterization than using a single visual cue. This system was validated under real-life fatigue conditions with human subjects of different ethnic backgrounds, genders, and ages; with/without glasses; and under different illumination conditions. It was found to be reasonably robust, reliable, and accurate in fatigue characterization.

[1]  Hiroshi Ueno,et al.  Development of drowsiness detection system , 1994, Proceedings of VNIS'94 - 1994 Vehicle Navigation and Information Systems Conference.

[2]  Mark R. Rosekind,et al.  Crew Factors in Flight Operations XI: A Survey of Fatigue Factors in Regional Airline Operations , 1999 .

[3]  Tai Sing Lee,et al.  Image Representation Using 2D Gabor Wavelets , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  J H Richardson.  The development of a driver alertness monitoring system , 1993 .

[5]  Qiang Ji,et al.  Real time 3D face pose discrimination based on active IR illumination , 2002, Object recognition supported by user interaction for service robots.

[6]  B. Carnahan,et al.  A drowsy driver detection system for heavy vehicles , 1998, 17th DASC. AIAA/IEEE/SAE. Digital Avionics Systems Conference. Proceedings (Cat. No.98CH36267).

[7]  Worthy N. Martin,et al.  Human-computer interaction using eye-gaze input , 1989, IEEE Trans. Syst. Man Cybern..

[8]  Irfan A. Essa,et al.  Detecting and tracking eyes by using their physiological properties, dynamics, and appearance , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[9]  Hiroshi Saito,et al.  Applications of driver's line of sight to automobiles-what can driver's eye tell , 1994, Proceedings of VNIS'94 - 1994 Vehicle Navigation and Information Systems Conference.

[10]  Yoshinobu Ebisawa,et al.  Unconstrained Pupil Detection TechniqueUsing Two Light Sources And The ImageDifference Method , 1970 .

[11]  D. Dinges,et al.  EVALUATION OF TECHNIQUES FOR OCULAR MEASUREMENT AS AN INDEX OF FATIGUE AND THE BASIS FOR ALERTNESS MANAGEMENT , 1998 .

[12]  Zhiwei Zhu,et al.  3D face pose tracking from an uncalibrated monocular camera , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[13]  Zhiwei Zhu,et al.  Combining Kalman filtering and mean shift for real time eye tracking under active IR illumination , 2002, Object recognition supported by user interaction for service robots.

[14]  Hiroshi Ueno,et al.  DEVELOPMENT OF A DROWSINESS WARNING SYSTEM , 1995 .

[15]  Mark R. Rosekind,et al.  Crew Factors in Flight Operations. 8; A Survey of Fatigue Factors in Corporate/Executive A Viation Operations , 2000 .

[16]  Zhiwei Zhu,et al.  Real-time eye detection and tracking under various light conditions , 2002, ETRA.

[17]  Margrit Betke,et al.  Preliminary investigation of real-time monitoring of a driver in city traffic , 2000, Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No.00TH8511).

[18]  Etienne Grandjean.  Fitting the task to the man , 1969 .

[19]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[20]  P H Baas,et al.  FATIGUE AND FITNESS FOR DUTY OF NEW ZEALAND TRUCK DRIVERS , 1998 .

[21]  Zhiwei Zhu,et al.  Eye and gaze tracking for interactive graphic display , 2002, SMARTGRAPH '02.

[22]  T. Åkerstedt,et al.  The three-process model of alertness and its extension to performance, sleep latency, and sleep length. , 1997, Chronobiology international.

[23]  S Saito,et al.  Does fatigue exist in a quantitative measurement of eye movements? , 1992, Ergonomics.

[24]  Zhiwei Zhu,et al.  Active facial tracking for fatigue detection , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[25]  N Mabbott,et al.  PROCEDURES AND DEVICES TO MONITOR OPERATOR ALERTNESS WHILST OPERATING MACHINERY IN OPEN-CUT MINES , 1999 .

[26]  Yoshinobu Ebisawa,et al.  Improved video-based eye-gaze detection method , 1994, Conference Proceedings. 10th Anniversary. IMTC/94. Advanced Technologies in I & M. 1994 IEEE Instrumentation and Measurement Technolgy Conference (Cat. No.94CH3424-9).

[27]  Tim Horberry,et al.  REVIEW OF FATIGUE DETECTION AND PREDICTION TECHNOLOGIES , 2000 .

[28]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[29]  P Sherry.  FATIGUE COUNTERMEASURES IN THE RAILROAD INDUSTRY: PAST AND CURRENT DEVELOPMENTS , 2000 .

[30]  C Tarriere,et al.  AN ON-BOARD SYSTEM FOR DETECTING LAPSES OF ALERTNESS IN CAR DRIVING , 1995 .

[31]  Donald F. Specht,et al.  A general regression neural network , 1991, IEEE Trans. Neural Networks.

[32]  D. Dawson,et al.  QUANTITATIVE SIMILARITY BETWEEN THE COGNITIVE PSYCHOMOTOR PERFORMANCE DECREMENT ASSOCIATED WITH SUSTAINED WAKEFULNESS AND ALCOHOL INTOXICATION , 1998 .

[33]  Finn V. Jensen,et al.  Bayesian Networks and Decision Graphs , 2001, Statistics for Engineering and Information Science.

[34]  R. Onken.  DAISY, an adaptive, knowledge-based driver monitoring and warning system , 1994, Proceedings of VNIS'94 - 1994 Vehicle Navigation and Information Systems Conference.

[35]  Qiang Ji,et al.  Real Time Visual Cues Extraction for Monitoring Driver Vigilance , 2001, ICVS.

[36]  S. Boverie,et al.  INTELLIGENT SYSTEM FOR VIDEO MONITORING OF VEHICLE COCKPIT , 1998 .

[37]  E. N. Corlett,et al.  BooksFitting the task to the man — An ergonomic approach (3rd Ed): E. Grandjean Taylor & Francis Ltd, London, pp 375 & ix, £14.50 (£9.50 paper) , 1980 .

[38]  Melissa M. Mallis,et al.  MANAGING FATIGUE BY DROWSINESS DETECTION: CAN TECHNOLOGICAL PROMISES BE REALIZED? , 1998 .

[39]  Nancy J. Wesensten,et al.  SUSTAINING PERFORMANCE DURING CONTINUOUS OPERATIONS: THE U.S. ARMY'S SLEEP MANAGEMENT SYSTEM , 1998 .

[40]  Arnon Amir,et al.  Framerate pupil detector and gaze tracker , 1999, ICCV 1999.

[41]  C Tarriere,et al.  RESULTS OF THE FEASIBILITY STUDY OF A SYSTEM FOR WARNING OF DROWSINESS AT THE STEERING WHEEL BASED ON ANALYSIS OF DRIVER EYELID MOVEMENTS , 1996 .

[42]  David,et al.  Pose Discriminiation and Eye Detection Using Support Vector Machines (SVM) , 1998 .

[43]  Naoki Mukawa,et al.  FreeGaze: a gaze tracking system for everyday gaze interaction , 2002, ETRA.