A Multimodal System for Assessing Alertness Levels Due to Cognitive Loading

This paper proposes a scheme for assessing the alertness levels of an individual using simultaneous acquisition of multimodal physiological signals and fusing the information into a single metric for quantification of alertness. The system takes electroencephalogram, high-speed image sequence, and speech data as inputs. Certain parameters are computed from each of these measures as indicators of alertness and a metric is proposed using a fusion of the parameters for indicating alertness level of an individual at an instant. The scheme has been validated experimentally using standard neuropsychological tests, such as the Visual Response Test (VRT), Auditory Response Test (ART), a Letter Counting (LC) task, and the Stroop Test. The tests are used both as cognitive tasks to induce mental fatigue as well as tools to gauge the present degree of alertness of the subject. Correlation between the measures has been studied and the experimental variables have been statistically analyzed using measures such as multivariate linear regression and analysis of variance. Correspondence of trends obtained from biomarkers and neuropsychological measures validate the usability of the proposed metric.

[1]  Yan Liu,et al.  A novel two-step SVM classifier for voiced/unvoiced/silence classification of speech , 2004, 2004 International Symposium on Chinese Spoken Language Processing.

[2]  Aurobinda Routray,et al.  Effect of Sleep Deprivation on Functional Connectivity of EEG Channels , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[3]  J. Horne,et al.  Vehicle accidents related to sleep: a review. , 1999, Occupational and environmental medicine.

[4]  Joel S. Warm,et al.  Selection for vigilance assignments: a review and proposed new direction , 2011 .

[5]  W Karwowski,et al.  The theory of the STOP-emotion: On the functionality of fatigue , 2000 .

[6]  M. Posner,et al.  Research on attention networks as a model for the integration of psychological science. , 2007, Annual review of psychology.

[7]  G. Matthews.  Human Performance: Cognition, Stress and Individual Differences , 2000 .

[8]  R. Barry,et al.  Removal of ocular artifact from the EEG: a review , 2000, Neurophysiologie Clinique/Clinical Neurophysiology.

[9]  Philipp Kanske,et al.  Probing the Mechanisms of Attention , 2007 .

[10]  Curtis S. Ikehara,et al.  Methodological issues of real time data acquisition from multiple sources of physiological data , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[11]  F. D. Silva.  Neural mechanisms underlying brain waves: from neural membranes to networks. , 1991 .

[12]  Tieniu Tan,et al.  Toward Accurate and Fast Iris Segmentation for Iris Biometrics , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Erhardt Barth,et al.  Accurate Eye Centre Localisation by Means of Gradients , 2011, VISAPP.

[14]  Aurobinda Routray,et al.  A Vision-Based System for Monitoring the Loss of Attention in Automotive Drivers , 2013, IEEE Transactions on Intelligent Transportation Systems.

[15]  A. Craig,et al.  Driver fatigue: electroencephalography and psychological assessment. , 2002, Psychophysiology.

[16]  Chris Berka,et al.  Drowsiness/alertness algorithm development and validation using synchronized EEG and cognitive performance to individualize a generalized model , 2011, Biological Psychology.

[17]  Kerstin Tham,et al.  The baking tray task: a test of spatial neglect. , 1996, Neuropsychological rehabilitation.

[18]  D. Dijk,et al.  Dose-response relationship for light intensity and ocular and electroencephalographic correlates of human alertness , 2000, Behavioural Brain Research.

[19]  Luzheng Bi,et al.  Study on Real-time Detection of Alertness Based on EEG , 2007, 2007 IEEE/ICME International Conference on Complex Medical Engineering.

[20]  S D Baulk,et al.  Countermeasures to driver fatigue: a review of public awareness campaigns and legal approaches , 2005, Australian and New Zealand journal of public health.

[21]  Mubarak Shah,et al.  Monitoring head/eye motion for driver alertness with one camera , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[22]  J. Lani,et al.  General Health Questionnaire (GHQ) , 2010 .

[23]  Masakiyo Fujimoto,et al.  Voice activity detection based on adjustable linear prediction and GARCH models , 2008, Speech Commun..

[24]  S. Kar,et al.  EEG signal analysis for the assessment and quantification of driver’s fatigue , 2010 .

[25]  Anjith George,et al.  An on-board vision based system for drowsiness detection in automotive drivers , 2013 .

[26]  K. Willmes,et al.  On the Functional Neuroanatomy of Intrinsic and Phasic Alertness , 2001, NeuroImage.

[27]  Aurobinda Routray,et al.  Study of Loss of Alertness and Driver Fatigue Using Visibility Graph Synchronization , 2016 .

[28]  Aurobinda Routray,et al.  A novel drowsiness detection scheme based on speech analysis with validation using simultaneous EEG recordings , 2010, 2010 IEEE International Conference on Automation Science and Engineering.

[29]  M. Ferrara,et al.  Slow eye movements and subjective estimates of sleepiness predict EEG power changes during sleep deprivation. , 2007, Sleep.

[30]  Michael E. Smith,et al.  Neurophysiological measures of cognitive workload during human-computer interaction , 2003 .

[31]  G.F. Inbar,et al.  An improved P300-based brain-computer interface , 2005, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[32]  Y. Uchikawa,et al.  Relation between human alertness, velocity wave profile of saccade, and performance of visual activities , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[33]  D. Dinges,et al.  Behavioral and physiological consequences of sleep restriction. , 2007, Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine.

[34]  Alain Muzet,et al.  Defining the states of consciousness , 2001, Neuroscience & Biobehavioral Reviews.

[35]  Lawrence R. Rabiner,et al.  A pattern recognition approach to voiced-unvoiced-silence classification with applications to speech recognition , 1976 .

[36]  Xiaoyang Tan,et al.  Enhanced Local Texture Feature Sets for Face Recognition Under Difficult Lighting Conditions , 2007, IEEE Transactions on Image Processing.

[37]  B. Oken,et al.  Vigilance, alertness, or sustained attention: physiological basis and measurement , 2006, Clinical Neurophysiology.

[38]  Joel S. Warm,et al.  Vigilance Requires Hard Mental Work and Is Stressful , 2008, Hum. Factors.

[39]  Peter Nickel,et al.  Sensitivity of candidate markers of psychophysiological strain to cyclical changes in manual control load during simulated process control. , 2009, Applied ergonomics.

[40]  W. Klimesch.  EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis , 1999, Brain Research Reviews.

[41]  J. Cacioppo,et al.  Handbook Of Psychophysiology , 2019 .

[42]  Aurobinda Routray,et al.  Estimation of Saccadic Ratio from eye image sequences to detect human alertness , 2012, 2012 4th International Conference on Intelligent Human Computer Interaction (IHCI).

[43]  Abdulhamit Subasi,et al.  Automatic recognition of alertness level by using wavelet transform and artificial neural network , 2004, Journal of Neuroscience Methods.

[44]  Aurobinda Routray,et al.  Complex brain networks using Visibility Graph synchronization , 2013, 2013 Annual IEEE India Conference (INDICON).

[45]  David F. Dinges,et al.  Microcomputer analyses of performance on a portable, simple visual RT task during sustained operations , 1985 .

[46]  J. Coull.  Neural correlates of attention and arousal: insights from electrophysiology, functional neuroimaging and psychopharmacology , 1998, Progress in Neurobiology.

[47]  J. Buhle,et al.  Typologies of attentional networks , 2006, Nature Reviews Neuroscience.

[48]  D. Dinges.  An overview of sleepiness and accidents , 1995, Journal of sleep research.

[49]  Philipos C. Loizou,et al.  Speech Enhancement: Theory and Practice , 2007 .

[50]  Charles J. Golden,et al.  Stroop Color and Word Test , 1978 .

[51]  T. Sejnowski,et al.  Estimating alertness from the EEG power spectrum , 1997, IEEE Transactions on Biomedical Engineering.

[52]  V. Krishnaveni,et al.  Removal of ocular artifacts from EEG using adaptive thresholding of wavelet coefficients , 2006, Journal of neural engineering.

[53]  Stephen J. Boies,et al.  Components of attention. , 1971 .

[54]  Ying Zheng,et al.  Semantic feature extraction for accurate eye corner detection , 2008, 2008 19th International Conference on Pattern Recognition.

[55]  Brian C J Moore,et al.  Effect of spatial separation, extended bandwidth, and compression speed on intelligibility in a competing-speech task. , 2010, The Journal of the Acoustical Society of America.

[56]  C Tarriere,et al.  AN ON-BOARD SYSTEM FOR DETECTING LAPSES OF ALERTNESS IN CAR DRIVING , 1995 .

[57]  J. Lyznicki,et al.  Sleepiness, driving, and motor vehicle crashes , 1998 .

[58]  Aurobinda Routray,et al.  Functional network changes associated with sleep deprivation and fatigue during simulated driving: Validation using blood biomarkers , 2011, Clinical Neurophysiology.