Recognition of Mental Workload Levels Under Complex Human–Machine Collaboration by Using Physiological Features and Adaptive Support Vector Machines

In order to detect human operator performance degradation or breakdown, this paper proposes an adaptive support vector machine-based method to classify operator mental workload (MWL) into few discrete levels based on psychophysiological measures. Electroencephalogram, electrocardiogram, and electrooculography signals were recorded continuously while the operator was performing safety-critical process control operations in a simulated human-machine system. In coarse-grained analysis, the adaptive exponential smoothing (AES) technique is used to smooth the psychophysiological data and to remove strong artifacts without requiring templates. The MWL level is classified every 30 s by using bounded support vector machine (BSVM) and tenfold cross-validation techniques. Locality preservation projection (LPP) technique is utilized to derive salient psychophysiological features by means of feature reduction. By combining the AES-LPP and BSVM methods, the accuracy of the coarse-grained MWL classification was significantly improved by 11-13%. On the other hand, to perform MWL classification with higher temporal resolution and cross-subject and cross-trial generalizability, finer-grained data analysis is also conducted to recognize MWL levels every 5 s based on a combination of adaptive BSVM (ABSVM) and AES techniques. In comparison with the use of the BSVM algorithm alone, a significant performance improvement by 10-20% is achieved by using the AES-ABSVM method in the finer-grained MWL classification.

[1]  Zhong Yin,et al.  Identification of temporal variations in mental workload using locally-linear-embedding-based EEG feature reduction and support-vector-machine-based clustering and classification techniques , 2014, Comput. Methods Programs Biomed..

[2]  Yong Chen,et al.  Detection of temporal changes in psychophysiological data using statistical process control methods , 2012, Comput. Methods Programs Biomed..

[3]  David B. Kaber,et al.  Out‐of‐the‐loop performance problems and the use of intermediate levels of automation for improved control system functioning and safety , 1997 .

[4]  D de Waard,et al.  Cardiovascular state changes during performance of a simulated ambulance dispatchers' task: potential use for adaptive support. , 2009, Applied ergonomics.

[5]  Zhong Yin,et al.  Operator functional state classification using least-square support vector machine based recursive feature elimination technique , 2014, Comput. Methods Programs Biomed..

[6]  George Panoutsos,et al.  Real-Time Adaptive Automation System Based on Identification of Operator Functional State in Simulated Process Control Operations , 2010, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[7]  Eugene L. Grant,et al.  Statistical Quality Control , 1946 .

[8]  Glenn F. Wilson,et al.  Topographical changes in the ongoing EEG related to the difficulty of mental tasks , 2005, Brain Topography.

[9]  H. Gr.  Compensatory control in the regulation of human performance under stress and high workload; a cognitive-energetical framework. , 1997 .

[10]  William Marnane,et al.  Multimodal detection of head-movement artefacts in EEG , 2013, Journal of Neuroscience Methods.

[11]  Eric O. Postma,et al.  Dimensionality Reduction: A Comparative Review , 2008 .

[12]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[13]  David B. Kaber,et al.  Situation awareness implications of adaptive automation for information processing in an air traffic control-related task , 2006 .

[14]  J. Gregory Trafton,et al.  Dynamic Operator Overload: A Model for Predicting Workload During Supervisory Control , 2014, IEEE Transactions on Human-Machine Systems.

[15]  Raja Parasuraman,et al.  Display Integration Enhances Information Sampling and Decision Making in Automated Fault Management in a Simulated Spaceflight Micro-World , 2002, Aviation, space, and environmental medicine.

[16]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[17]  Shaozeng Yang,et al.  An adaptive human-machine control system based on multiple fuzzy predictive models of operator functional state , 2013, Biomed. Signal Process. Control..

[18]  Glenn F. Wilson,et al.  Performance Enhancement in an Uninhabited Air Vehicle Task Using Psychophysiologically Determined Adaptive Aiding , 2007, Hum. Factors.

[19]  Kenneth W. Bauer,et al.  Improving pilot mental workload classification through feature exploitation and combination: a feasibility study , 2005, Comput. Oper. Res..

[20]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[21]  M R Endsley,et al.  Level of automation effects on performance, situation awareness and workload in a dynamic control task. , 1999, Ergonomics.

[22]  R. Parasuraman,et al.  Psychophysiology and adaptive automation , 1996, Biological Psychology.

[23]  Pavlo A. Krokhmal,et al.  An algorithm for online detection of temporal changes in operator cognitive state using real-time psychophysiological data , 2010, Biomed. Signal Process. Control..

[24]  Jörg Raisch,et al.  Predictive modeling of human operator cognitive state via sparse and robust support vector machines , 2013, Cognitive Neurodynamics.

[25]  Peter Nickel,et al.  Sensitivity and Diagnosticity of the 0.1-Hz Component of Heart Rate Variability as an Indicator of Mental Workload , 2003, Hum. Factors.

[26]  Glenn F. Wilson,et al.  Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural Networks , 2003, Hum. Factors.

[27]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[28]  David B. Kaber,et al.  Emotional State Classification in Patient–Robot Interaction Using Wavelet Analysis and Statistics-Based Feature Selection , 2013, IEEE Transactions on Human-Machine Systems.

[29]  Giovanna Capizzi,et al.  An Adaptive Exponentially Weighted Moving Average Control Chart , 2003, Technometrics.

[30]  Qiang Ji,et al.  Cross-subject workload classification with a hierarchical Bayes model , 2012, NeuroImage.

[31]  Chih-Jen Lin,et al.  A Simple Decomposition Method for Support Vector Machines , 2002, Machine Learning.

[32]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[33]  J. Kamiya,et al.  A simple on-line technique for removing eye movement artifacts from the EEG. , 1973, Electroencephalography and clinical neurophysiology.

[34]  Lloyd S. Nelson.  Statistical Quality Control, 6th Ed. , 1991 .

[35]  Adam W. Hoover,et al.  Real-time detection of workload changes using heart rate variability , 2012, Biomed. Signal Process. Control..

[36]  A. Kramer,et al.  Physiological metrics of mental workload: A review of recent progress , 1990, Multiple-task performance.

[37]  James C. Christensen,et al.  The effects of day-to-day variability of physiological data on operator functional state classification , 2012, NeuroImage.

[38]  A.W.K. Gaillard,et al.  Operator Functional State: The Assessment and Prediction of Human Performance Degradation in Complex Tasks , 2003 .

[39]  Wei Jia,et al.  Locality preserving discriminant projections for face and palmprint recognition , 2010, Neurocomputing.

[40]  G. R. J. Hockey.  Compensatory control in the regulation of human performance under stress and high workload: A cognitive-energetical framework , 1997, Biological Psychology.

[41]  Jianhua Zhang,et al.  Support vector machine approaches to classifying operator functional state in human-machine system , 2011, Proceedings of the 30th Chinese Control Conference.

[42]  W. Brent Seales,et al.  Measuring Surgeons’ Mental Workload With a Time-Based Secondary Task , 2013 .

[43]  J Sauer,et al.  A conceptual framework for designing micro-worlds for complex work domains: a case study of the Cabin Air Management System. , 2000, Computers in human behavior.

[44]  P. Rousseeuw.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[45]  Xingyu Wang,et al.  ssessment of human operator functional state using a novel differential volution optimization based adaptive fuzzy model , 2011 .