Imbalanced classification of mental workload using a cost-sensitive majority weighted minority oversampling strategy

Identifying the temporal variations in mental workload level (MWL) is crucial for enhancing the safety of human–machine system operations, especially when there is cognitive overload or inattention of human operator. This paper proposed a cost-sensitive majority weighted minority oversampling strategy to address the imbalanced MWL data classification problem. Both the inter-class and intra-class imbalance problems are considered. For the former, imbalance ratio is defined to determine the number of the synthetic samples in the minority class. The latter problem is addressed by assigning different weights to borderline samples in the minority class based on the distance and density meaures of the sample distribution. Furthermore, multi-label classifier is designed based on an ensemble of binary classifiers. The results of analyzing 21 imbalanced UCI multi-class datasets showed that the proposed approach can effectively cope with the imbalanced classification problem in terms of several performance metrics including geometric mean (G-mean) and average accuracy (ACC). Moreover, the proposed approach was applied to the analysis of the EEG data of eight experimental participants subject to fluctuating levels of mental workload. The comparative results showed that the proposed method provides a competing alternative to several existing imbalanced learning algorithms and significantly outperforms the basic/referential method that ignores the imbalance nature of the dataset.

[1]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[2]  Jacek M. Zurada,et al.  Training neural network classifiers for medical decision making: The effects of imbalanced datasets on classification performance , 2008, Neural Networks.

[3]  Zhi-Hua Zhou,et al.  Exploratory Undersampling for Class-Imbalance Learning , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[4]  Bartosz Krawczyk,et al.  Cost-Sensitive Neural Network with ROC-Based Moving Threshold for Imbalanced Classification , 2015, IDEAL.

[5]  J. Kamiya,et al.  A simple on-line technique for removing eye movement artifacts from the EEG. , 1973, Electroencephalography and clinical neurophysiology.

[6]  Kathryn Mearns,et al.  Situation awareness and safety in offshore drill crews , 2006, Cognition, Technology & Work.

[7]  James C. Christensen,et al.  Deep long short-term memory structures model temporal dependencies improving cognitive workload estimation , 2017, Pattern Recognit. Lett..

[8]  Andrew L. Kun,et al.  Estimating cognitive load using remote eye tracking in a driving simulator , 2010, ETRA.

[9]  Tom Kontogiannis,et al.  Strategies in controlling, coordinating and adapting performance in air traffic control: modelling ‘loss of control’ events , 2011, Cognition, Technology & Work.

[10]  Gunwoo Kim,et al.  Classification cost: An empirical comparison among traditional classifier, Cost-Sensitive Classifier, and MetaCost , 2012, Expert Syst. Appl..

[11]  Rubin Wang,et al.  Recognition of Mental Workload Levels Under Complex Human–Machine Collaboration by Using Physiological Features and Adaptive Support Vector Machines , 2015, IEEE Transactions on Human-Machine Systems.

[12]  Zhi-Hua Zhou,et al.  The Influence of Class Imbalance on Cost-Sensitive Learning: An Empirical Study , 2006, Sixth International Conference on Data Mining (ICDM'06).

[13]  Richard J. Holden,et al.  Cognitive performance-altering effects of electronic medical records: an application of the human factors paradigm for patient safety , 2011, Cognition, Technology & Work.

[14]  H. Jasper Report of the committee on methods of clinical examination in electroencephalography , 1958 .

[15]  Fulko van Westrenen Modelling arrival control in a vessel traffic management system , 2014, Cognition, Technology & Work.

[16]  Xin Yao,et al.  MWMOTE--Majority Weighted Minority Oversampling Technique for Imbalanced Data Set Learning , 2014 .

[17]  Ellen M. Voorhees,et al.  Implementing agglomerative hierarchic clustering algorithms for use in document retrieval , 1986, Inf. Process. Manag..

[18]  Yang Wang,et al.  Boosting for Learning Multiple Classes with Imbalanced Class Distribution , 2006, Sixth International Conference on Data Mining (ICDM'06).

[19]  Haibo He,et al.  ADASYN: Adaptive synthetic sampling approach for imbalanced learning , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[20]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[21]  Haibo He,et al.  Learning from Imbalanced Data , 2009, IEEE Transactions on Knowledge and Data Engineering.

[22]  Pedro M. Domingos MetaCost: a general method for making classifiers cost-sensitive , 1999, KDD '99.

[23]  Tom Kontogiannis,et al.  Training Effective Human Performance in the Management of Stressful Emergencies , 1999, Cognition, Technology & Work.

[24]  Robert Oostenveld,et al.  Estimating workload using EEG spectral power and ERPs in the n-back task , 2012, Journal of neural engineering.

[25]  Claudia V. Goldman,et al.  On human–machine relations , 2017, Cognition, Technology & Work.

[26]  Mohammad Zulkernine,et al.  A hybrid network intrusion detection technique using random forests , 2006, First International Conference on Availability, Reliability and Security (ARES'06).

[27]  Andrew K. C. Wong,et al.  Classification of Imbalanced Data: a Review , 2009, Int. J. Pattern Recognit. Artif. Intell..

[28]  Nitesh V. Chawla,et al.  Editorial: special issue on learning from imbalanced data sets , 2004, SKDD.

[29]  Hui Han,et al.  Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning , 2005, ICIC.

[30]  I. Tomek,et al.  Two Modifications of CNN , 1976 .

[31]  Maaria Nuutinen,et al.  Expert Identity construct in analysing prerequisites for expertise development: a case study of nuclear power plant operators’ on-the-job training , 2005, Cognition, Technology & Work.

[32]  Min Chen,et al.  Deep Learning for Imbalanced Multimedia Data Classification , 2015, 2015 IEEE International Symposium on Multimedia (ISM).

[33]  Maarten A. Hogervorst,et al.  Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload , 2014, Front. Neurosci..

[34]  Francisco Herrera,et al.  Evolutionary-based selection of generalized instances for imbalanced classification , 2012, Knowl. Based Syst..

[35]  R. Parasuraman,et al.  Psychophysiology and adaptive automation , 1996, Biological Psychology.

[36]  Brett J. Borghetti,et al.  A New Feature for Cross-Day Psychophysiological Workload Estimation , 2016, 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA).

[37]  Longbing Cao,et al.  Training deep neural networks on imbalanced data sets , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[38]  M. Maloof Learning When Data Sets are Imbalanced and When Costs are Unequal and Unknown , 2003 .

[39]  Yung-Tsan Jou,et al.  Evaluation of operators’ mental workload of human–system interface automation in the advanced nuclear power plants , 2009 .

[40]  A.W.K. Gaillard,et al.  Operator Functional State: The Assessment and Prediction of Human Performance Degradation in Complex Tasks , 2003 .

[41]  Mohammad Zulkernine,et al.  Network Intrusion Detection using Random Forests , 2005, PST.

[42]  Edit Varga,et al.  Manipulation of mental models of anatomy in interventional radiology and its consequences for design of human–computer interaction , 2013, Cognition, Technology & Work.

[43]  Kaustubh Manik Gaikwad,et al.  Removal of high frequency noise from ECG signal using digital IIR butterworth filter , 2014, 2014 IEEE Global Conference on Wireless Computing & Networking (GCWCN).

[44]  Tarja Susi,et al.  Technostress in the office: a distributed cognition perspective on human–technology interaction , 2013, Cognition, Technology & Work.

[45]  Charles Elkan,et al.  The Foundations of Cost-Sensitive Learning , 2001, IJCAI.

[46]  Marco Vannucci,et al.  A method for resampling imbalanced datasets in binary classification tasks for real-world problems , 2014, Neurocomputing.

[47]  Haibo He,et al.  RAMOBoost: Ranked Minority Oversampling in Boosting , 2010, IEEE Transactions on Neural Networks.

[48]  Igor Kononenko,et al.  Cost-Sensitive Learning with Neural Networks , 1998, ECAI.

[49]  Zhi-Hua Zhou,et al.  Ieee Transactions on Knowledge and Data Engineering 1 Training Cost-sensitive Neural Networks with Methods Addressing the Class Imbalance Problem , 2022 .

[50]  Jürgen S. Sauer,et al.  Effects of Sleep Deprivation and User Interface on Complex Performance: A Multilevel Analysis of Compensatory Control , 1998, Hum. Factors.

[51]  Hasan Ayaz,et al.  Optical brain monitoring for operator training and mental workload assessment , 2012, NeuroImage.

[52]  H. Jasper,et al.  The ten-twenty electrode system of the International Federation. The International Federation of Clinical Neurophysiology. , 1999, Electroencephalography and clinical neurophysiology. Supplement.

[53]  Gang Wang Asymmetric Random Subspace Method for Imbalanced Credit Risk Evaluation , 2012 .

[54]  Glenn F. Wilson,et al.  Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural Networks , 2003, Hum. Factors.

[55]  A. Craig,et al.  A critical review of the psychophysiology of driver fatigue , 2001, Biological Psychology.

[56]  Xin Yao,et al.  Multiclass Imbalance Problems: Analysis and Potential Solutions , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[57]  Xin Yao,et al.  Dynamic Sampling Approach to Training Neural Networks for Multiclass Imbalance Classification , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[58]  Xin Yao,et al.  Ieee Transactions on Knowledge and Data Engineering 1 Relationships between Diversity of Classification Ensembles and Single-class Performance Measures , 2022 .

[59]  Hinrich Schütze,et al.  Projections for efficient document clustering , 1997, SIGIR '97.

[60]  Zhi-Hua Zhou,et al.  Exploratory Under-Sampling for Class-Imbalance Learning , 2006, Sixth International Conference on Data Mining (ICDM'06).