Enhanced Drowsiness Detection Using Deep Learning: An fNIRS Study

In this paper, a deep-learning-based driver-drowsiness detection for brain-computer interface (BCI) using functional near-infrared spectroscopy (fNIRS) is investigated. The passive brain signals from drowsiness were acquired from 13 healthy subjects while driving a car simulator. The brain activities were measured with a continuous-wave fNIRS system, in which the prefrontal and dorsolateral prefrontal cortices were focused. Deep neural networks (DNN) were pursued to classify the drowsy and alert states. For training and testing the models, the convolutional neural networks (CNN) were used on color map images to determine the best suitable channels for brain activity detection in 0~1, 0~3, 0~5, and 0~10 second time windows. The average accuracies (i.e., 82.7, 89.4, 93.7, and 97.2% in the 0~1, 0~3, 0~5, and 0~10 sec time windows, respectively) using DNNs from the right dorsolateral prefrontal cortex were obtained. The CNN architecture resulted in an average accuracy of 99.3%, showing the model to be capable of differentiating the images of drowsy/non-drowsy states. The proposed approach is promising for detecting drowsiness and in accessing the brain location for a passive BCI.

[1]  Yves Rosseel,et al.  A Review of fMRI Simulation Studies , 2014, PloS one.

[2]  Frédéric Dehais,et al.  Detecting Pilot's Engagement Using fNIRS Connectivity Features in an Automated vs. Manual Landing Scenario , 2018, Front. Hum. Neurosci..

[3]  Yan Su,et al.  Deep learning for in vitro prediction of pharmaceutical formulations , 2018, Acta pharmaceutica Sinica. B.

[4]  Keum-Shik Hong,et al.  Multivariable Adaptive Control of the Rewinding Process of a Roll-to-roll System Governed by Hyperbolic Partial Differential Equations , 2018, International Journal of Control, Automation and Systems.

[5]  Jun Li,et al.  Unsupervised Feature Extraction in Hyperspectral Images Based on Wasserstein Generative Adversarial Network , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[6]  Francisco Herrera,et al.  A binocular image fusion approach for minimizing false positives in handgun detection with deep learning , 2019, Inf. Fusion.

[7]  Pablo Laguna,et al.  Drowsiness detection using heart rate variability , 2016, Medical & Biological Engineering & Computing.

[8]  N. Logothetis What we can do and what we cannot do with fMRI , 2008, Nature.

[9]  Amad Zafar,et al.  Neuronal Activation Detection Using Vector Phase Analysis with Dual Threshold Circles: A Functional Near-Infrared Spectroscopy Study , 2018, Int. J. Neural Syst..

[10]  Christian Kothe,et al.  Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general , 2011, Journal of neural engineering.

[11]  S. Coyle,et al.  Brain–computer interfaces: a review , 2003 .

[12]  Sangtae Ahn,et al.  Exploring Neuro-Physiological Correlates of Drivers' Mental Fatigue Caused by Sleep Deprivation Using Simultaneous EEG, ECG, and fNIRS Data , 2016, Front. Hum. Neurosci..

[13]  Keum-Shik Hong,et al.  fNIRS-based brain-computer interfaces: a review , 2015, Front. Hum. Neurosci..

[14]  Matthias Scheutz,et al.  What we can and cannot (yet) do with functional near infrared spectroscopy , 2014, Front. Neurosci..

[15]  Jens Steinbrink,et al.  Decoding Vigilance with NIRS , 2014, PloS one.

[16]  Tao Zhang,et al.  Bayesian Nonnegative CP Decomposition-Based Feature Extraction Algorithm for Drowsiness Detection , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[17]  Ardalan Aarabi,et al.  Autoregressive model based algorithm for correcting motion and serially correlated errors in fNIRS. , 2013, Biomedical optics express.

[18]  Y. Kim,et al.  Classification of prefrontal and motor cortex signals for three-class fNIRS–BCI , 2015, Neuroscience Letters.

[19]  Tao Zhang,et al.  Drowsiness Detection by Bayesian-Copula Discriminant Classifier Based on EEG Signals During Daytime Short Nap , 2017, IEEE Transactions on Biomedical Engineering.

[20]  Simon G Hosking,et al.  Predicting driver drowsiness using vehicle measures: recent insights and future challenges. , 2009, Journal of safety research.

[21]  Keum-Shik Hong,et al.  Real-time feature extraction of P300 component using adaptive nonlinear principal component analysis , 2011, Biomedical engineering online.

[22]  Keum-Shik Hong,et al.  Passive BCI based on drowsiness detection: an fNIRS study. , 2015, Biomedical optics express.

[23]  M. Shamim Hossain,et al.  Emotion recognition using deep learning approach from audio-visual emotional big data , 2019, Inf. Fusion.

[24]  Tom Chau,et al.  Development of a Ternary Near-Infrared Spectroscopy Brain-Computer Interface: Online Classification of Verbal Fluency Task, Stroop Task and Rest , 2017, Int. J. Neural Syst..

[25]  Teresa Wilcox,et al.  fNIRS in the developmental sciences. , 2015, Wiley interdisciplinary reviews. Cognitive science.

[26]  Sanghoon Lee,et al.  Deep Visual Saliency on Stereoscopic Images , 2019, IEEE Transactions on Image Processing.

[27]  Keum-Shik Hong,et al.  Noise reduction in functional near-infrared spectroscopy signals by independent component analysis. , 2013, The Review of scientific instruments.

[28]  Licheng Jiao,et al.  Hyperspectral imagery classification with deep metric learning , 2019, Neurocomputing.

[29]  Hongwei Liu,et al.  Deep Max-Margin Discriminant Projection , 2019, IEEE Transactions on Cybernetics.

[30]  M. R. Bhutta,et al.  Note: three wavelengths near-infrared spectroscopy system for compensating the light absorbance by water. , 2014, The Review of scientific instruments.

[31]  M. V. D. Heuvel,et al.  Exploring the brain network: A review on resting-state fMRI functional connectivity , 2010, European Neuropsychopharmacology.

[32]  Wesley B. Baker,et al.  Modified Beer-Lambert law for blood flow , 2014, Photonics West - Biomedical Optics.

[33]  A. Craig,et al.  A critical review of the psychophysiology of driver fatigue , 2001, Biological Psychology.

[34]  Thibault Gateau,et al.  In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI , 2018, Front. Hum. Neurosci..

[35]  Tarek Sayed,et al.  Automated Analysis of Pedestrian–Vehicle Conflicts Using Video Data , 2009 .

[36]  Keum-Shik Hong,et al.  Online binary decision decoding using functional near-infrared spectroscopy for the development of brain–computer interface , 2014, Experimental Brain Research.

[37]  M. Doppelmayr,et al.  Current State and Future Prospects of EEG and fNIRS in Robot-Assisted Gait Rehabilitation: A Brief Review , 2019, Front. Hum. Neurosci..

[38]  Mobyen Uddin Ahmed,et al.  Automatic driver sleepiness detection using EEG, EOG and contextual information , 2019, Expert Syst. Appl..

[39]  Keum-Shik Hong,et al.  Single-trial lie detection using a combined fNIRS-polygraph system , 2015, Front. Psychol..

[40]  Qiang Ji,et al.  Real-Time Eye, Gaze, and Face Pose Tracking for Monitoring Driver Vigilance , 2002, Real Time Imaging.

[41]  Tzyy-Ping Jung,et al.  Toward Drowsiness Detection Using Non-hair-Bearing EEG-Based Brain-Computer Interfaces , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[42]  Yuan-Pin Lin,et al.  Independent Component Ensemble of EEG for Brain–Computer Interface , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[43]  Tzyy-Ping Jung,et al.  Spatial Filtering for EEG-Based Regression Problems in Brain–Computer Interface (BCI) , 2017, IEEE Transactions on Fuzzy Systems.

[44]  N. Birbaumer,et al.  Lower Limb Movement Preparation in Chronic Stroke , 2014, Neurorehabilitation and neural repair.

[45]  Robert C. Whitaker,et al.  Drowsy Driving, Sleep Duration, and Chronotype in Adolescents , 2019, The Journal of pediatrics.

[46]  Jae Gwan Kim,et al.  Utilization of a combined EEG/NIRS system to predict driver drowsiness , 2017, Scientific Reports.

[47]  Klaus-Robert Müller,et al.  Introduction to machine learning for brain imaging , 2011, NeuroImage.

[48]  M. Shamim Hossain,et al.  Deep Feature Learning for Disease Risk Assessment Based on Convolutional Neural Network With Intra-Layer Recurrent Connection by Using Hospital Big Data , 2018, IEEE Access.

[49]  Robert Riener,et al.  Detection of motor execution using a hybrid fNIRS-biosignal BCI: a feasibility study , 2013, Journal of NeuroEngineering and Rehabilitation.

[50]  Rajarathnam Chandramouli,et al.  Decoding Asynchronous Reaching in Electroencephalography Using Stacked Autoencoders , 2018, IEEE Access.

[51]  Lina Yao,et al.  A Survey on Deep Learning based Brain Computer Interface: Recent Advances and New Frontiers , 2019, ArXiv.

[52]  Marco Ferrari,et al.  A brief review on the history of human functional near-infrared spectroscopy (fNIRS) development and fields of application , 2012, NeuroImage.

[53]  Yue Wu,et al.  DeepDetect: A Cascaded Region-Based Densely Connected Network for Seismic Event Detection , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[54]  Hasan Ayaz,et al.  Speech Recognition via fNIRS Based Brain Signals , 2018, Front. Neurosci..

[55]  M. Toichi,et al.  Dorsolateral prefrontal cortical oxygenation during REM sleep in humans , 2011, Brain Research.

[56]  Jun Ma,et al.  Deep auto-encoder observer multiple-model fast aircraft actuator fault diagnosis algorithm , 2017, International Journal of Control, Automation and Systems.

[57]  Jun Li,et al.  Temporal correlation of spontaneous hemodynamic activity in language areas measured with functional near-infrared spectroscopy. , 2014, Biomedical optics express.

[58]  Toshihiro Hiraoka,et al.  Heart Rate Variability-Based Driver Drowsiness Detection and Its Validation With EEG , 2019, IEEE Transactions on Biomedical Engineering.

[59]  Hong-Hyun Kim,et al.  Multi-task convolutional neural network system for license plate recognition , 2017, International Journal of Control, Automation and Systems.

[60]  Keum-Shik Hong,et al.  Decoding Answers to Four-Choice Questions Using Functional near Infrared Spectroscopy , 2015 .

[61]  Larissa C Schudlo,et al.  Development and testing an online near-infrared spectroscopy brain–computer interface tailored to an individual with severe congenital motor impairments , 2018, Disability and rehabilitation. Assistive technology.

[62]  C. George,et al.  Sleep apnea, alertness, and motor vehicle crashes. , 2007, American journal of respiratory and critical care medicine.

[63]  David A. Boas,et al.  Twenty years of functional near-infrared spectroscopy: introduction for the special issue , 2014, NeuroImage.

[64]  Jiali Li,et al.  Design of a Fatigue Detection System for High-Speed Trains Based on Driver Vigilance Using a Wireless Wearable EEG , 2017, Sensors.