Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review

Considerable progress has been made in improving the estimation accuracy of cognitive workload using various sensor technologies. However, the overall performance of different algorithms and methods remain suboptimal in real-world applications. Some studies in the literature demonstrate that a single modality is sufficient to estimate cognitive workload. These studies are limited to controlled settings, a scenario that is significantly different from the real world where data gets corrupted, interrupted, and delayed. In such situations, the use of multiple modalities is needed. Multimodal fusion approaches have been successful in other domains, such as wireless-sensor networks, in addressing single-sensor weaknesses and improving information quality/accuracy. These approaches are inherently more reliable when a data source is lost. In the cognitive workload literature, sensors, such as electroencephalography (EEG), electrocardiography (ECG), and eye tracking, have shown success in estimating the aspects of cognitive workload. Multimodal approaches that combine data from several sensors together can be more robust for real-time measurement of cognitive workload. In this article, we review the published studies related to multimodal data fusion to estimate the cognitive workload and synthesize their main findings. We identify the opportunities for designing better multimodal fusion systems for cognitive workload modeling.

[1]  H. Hotelling Relations Between Two Sets of Variates , 1936 .

[2]  Hussein A. Abbass,et al.  Trusted Autonomy and Cognitive Cyber Symbiosis: Open Challenges , 2015, Cognitive Computation.

[3]  Glenn F. Wilson,et al.  Performance Enhancement in an Uninhabited Air Vehicle Task Using Psychophysiologically Determined Adaptive Aiding , 2007, Hum. Factors.

[4]  J. Beatty,et al.  Pupillometric signs of brain activation vary with level of cognitive processing. , 1978, Science.

[5]  S. Iyengar,et al.  Multi-Sensor Fusion: Fundamentals and Applications With Software , 1997 .

[6]  Wei You,et al.  Feature Weight Driven Interactive Mutual Information Modeling for Heterogeneous Bio-Signal Fusion to Estimate Mental Workload , 2017, Sensors.

[7]  Klaus-Robert Müller,et al.  Enhanced Performance by a Hybrid Nirs–eeg Brain Computer Interface , 2022 .

[8]  Andrew Sears,et al.  Automated stress detection using keystroke and linguistic features: An exploratory study , 2009, Int. J. Hum. Comput. Stud..

[9]  L. Mulder,et al.  Please Scroll down for Article Ergonomics Cardiovascular and Eye Activity Measures as Indices for Momentary Changes in Mental Effort during Simulated Flight Cardiovascular and Eye Activity Measures as Indices for Momentary Changes in Mental Effort during Simulated Flight , 2022 .

[10]  Jiankun Hu,et al.  Continuous authentication using EEG and face images for trusted autonomous systems , 2016, 2016 14th Annual Conference on Privacy, Security and Trust (PST).

[11]  T. Jung,et al.  Task performance and eye activity: predicting behavior relating to cognitive workload. , 2007, Aviation, space, and environmental medicine.

[12]  Brian P. Bailey,et al.  Categories & Subject Descriptors: H.5.2 [Information , 2022 .

[13]  Christopher D. Wickens,et al.  Multiple resources and performance prediction , 2002 .

[14]  Antonio Franchi,et al.  Human-Centered Design and Evaluation of Haptic Cueing for Teleoperation of Multiple Mobile Robots , 2013, IEEE Transactions on Cybernetics.

[15]  Dominic Heger,et al.  Hybrid fNIRS-EEG based discrimination of 5 levels of memory load , 2015, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER).

[16]  Rónán Collins,et al.  Combining fNIRS and EEG to Improve Motor Cortex Activity Classification during an Imagined Movement-Based Task , 2011, HCI.

[17]  Soo-Young Lee,et al.  EEG-Based Classification of Implicit Intention During Self-Relevant Sentence Reading , 2016, IEEE Transactions on Cybernetics.

[18]  James Llinas,et al.  Handbook of Multisensor Data Fusion : Theory and Practice, Second Edition , 2008 .

[19]  Glenn F. Wilson,et al.  An Analysis of Mental Workload in Pilots During Flight Using Multiple Psychophysiological Measures , 2002 .

[20]  Mohan M. Trivedi,et al.  Audiovisual Information Fusion in Human–Computer Interfaces and Intelligent Environments: A Survey , 2010, Proceedings of the IEEE.

[21]  Hussein A. Abbass,et al.  Augmented Cognition using Real-time EEG-based Adaptive Strategies for Air Traffic Control , 2014 .

[22]  Jing Qiu,et al.  Body Posture as an Indicator of Workload in Mental Work , 2012, Hum. Factors.

[23]  Siyuan Chen,et al.  Efficient and Robust Pupil Size and Blink Estimation From Near-Field Video Sequences for Human–Machine Interaction , 2014, IEEE Transactions on Cybernetics.

[24]  L. Mulder,et al.  CARDIOVASCULAR REACTIVITY AND MENTAL WORKLOAD , 1989 .

[25]  J. R. Comstock MAT - MULTI-ATTRIBUTE TASK BATTERY FOR HUMAN OPERATOR WORKLOAD AND STRATEGIC BEHAVIOR RESEARCH , 1994 .

[26]  Javier Del Ser,et al.  Cognitive workload classification using eye-tracking and EEG data , 2016, HCI-Aero.

[27]  Joonwoo Son,et al.  Identification of driver cognitive workload using support vector machines with driving performance, physiology and eye movement in a driving simulator , 2013 .

[28]  Yunfa Fu,et al.  A hybrid BCI based on EEG and fNIRS signals improves the performance of decoding motor imagery of both force and speed of hand clenching , 2015, Journal of neural engineering.

[29]  Mansi Sharma,et al.  Workload regulation by Sudarshan Kriya: an EEG and ECG perspective , 2016, Brain Informatics.

[30]  Iven Van Mechelen,et al.  A generic linked-mode decomposition model for data fusion , 2010 .

[31]  Sascha Meudt,et al.  Fusion Architectures for Multimodal Cognitive Load Recognition , 2016, MPRSS.

[32]  Christian Jutten,et al.  Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects , 2015, Proceedings of the IEEE.

[33]  Ulf Ahlstrom,et al.  Using eye movement activity as a correlate of cognitive workload , 2006 .

[34]  Tanja Schultz,et al.  Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS , 2014, Front. Hum. Neurosci..

[35]  Anima Majumder,et al.  Automatic Facial Expression Recognition System Using Deep Network-Based Data Fusion , 2018, IEEE Transactions on Cybernetics.

[36]  Mohammad Soleymani,et al.  Short-term emotion assessment in a recall paradigm , 2009, Int. J. Hum. Comput. Stud..

[37]  B. Kollmeier,et al.  Comparing Eye Tracking with Electrooculography for Measuring Individual Sentence Comprehension Duration , 2016, PloS one.

[38]  Guillaume Chanel,et al.  Emotion Assessment: Arousal Evaluation Using EEG's and Peripheral Physiological Signals , 2006, MRCS.

[39]  Andrzej Cichocki,et al.  EmotionMeter: A Multimodal Framework for Recognizing Human Emotions , 2019, IEEE Transactions on Cybernetics.

[40]  Ioan Opris,et al.  Neural correlates of fast pupil dilation in nonhuman primates: Relation to behavioral performance and cognitive workload , 2010, Behavioural Brain Research.

[41]  Stelios C. A. Thomopoulos Sensor integration and data fusion , 1990, J. Field Robotics.

[42]  Bernd Johannes,et al.  Author's Personal Copy Biological Psychology a Methodology to Compensate for Individual Differences in Psychophysiological Assessment , 2022 .

[43]  J. Roy,et al.  Data fusion concepts applied to a suite of dissimilar sensors , 1996, Proceedings of 1996 Canadian Conference on Electrical and Computer Engineering.

[44]  P. Shewokis,et al.  Mental workload classification with concurrent electroencephalography and functional near-infrared spectroscopy , 2017 .

[45]  Robert J. K. Jacob,et al.  Combining Electroencephalograph and Functional Near Infrared Spectroscopy to Explore Users' Mental Workload , 2009, HCI.

[46]  M. Pomplun,et al.  Pupil Dilation as an Indicator of Cognitive Workload in Human-Computer Interaction , 2003 .

[47]  Tanja Schultz,et al.  Multimodal Recognition of Cognitive Workload for Multitasking in the Car , 2010, 2010 20th International Conference on Pattern Recognition.

[48]  Hasan Ayaz,et al.  Multisubject “Learning” for Mental Workload Classification Using Concurrent EEG, fNIRS, and Physiological Measures , 2017, Front. Hum. Neurosci..

[49]  R. Vlek,et al.  Combined EEG-fNIRS Decoding of Motor Attempt and Imagery for Brain Switch Control: An Offline Study in Patients With Tetraplegia , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[50]  Tanja Schultz,et al.  Hybrid fNIRS-EEG based classification of auditory and visual perception processes , 2014, Front. Neurosci..

[51]  Daphne N. Yu,et al.  High-resolution EEG mapping of cortical activation related to working memory: effects of task difficulty, type of processing, and practice. , 1997, Cerebral cortex.

[52]  Magdalena Borys,et al.  Classifying Cognitive Workload Using Eye Activity and EEG Features in Arithmetic Tasks , 2017, ICIST.

[53]  Gerhard Tröster,et al.  EyeMote - Towards Context-Aware Gaming Using Eye Movements Recorded from Wearable Electrooculography , 2008, Fun and Games.

[54]  Michael P. Jenkins,et al.  Are Behavioral Measures Useful for Detecting Cognitive Workload During Human-Computer Interaction? , 2017 .

[55]  J. Veltman,et al.  Physiological indices of workload in a simulated flight task , 1996, Biological Psychology.

[56]  Denise M. Nicholson,et al.  Assessing Cognitive State with Multiple Physiological Measures: A Modular Approach , 2009, HCI.

[57]  A. H. Roscoe Assessing pilot workload. Why measure heart rate, HRV and respiration? , 1992, Biological Psychology.

[58]  Andreas Dunser,et al.  Combining EEG with Pupillometry to Improve Cognitive Workload Detection , 2015, Computer.

[59]  Anne-Marie Brouwer,et al.  Measuring workload using a combination of electroencephalography and near infrared spectroscopy , 2012 .

[60]  Gregory Richard Frank,et al.  Monitoring Seated Postural Responses to Assess Cognitive State , 2007 .

[61]  Elke Vlemincx,et al.  Individual differences in cardiorespiratory measures of mental workload: An investigation of negative affectivity and cognitive avoidant coping in pilot candidates. , 2017, Applied ergonomics.

[62]  T. Troscianko,et al.  Effort during visual search and counting: Insights from pupillometry , 2007, Quarterly journal of experimental psychology.

[63]  M.H. Moradi,et al.  Emotion detection using brain and peripheral signals , 2008, 2008 Cairo International Biomedical Engineering Conference.

[64]  Xu Huang,et al.  fNIRS Approach to Pain Assessment for Non-verbal Patients , 2017, ICONIP.

[65]  R. Calvo,et al.  Classification of Cognitive Load from Task Performance & Multichannel Physiology during Affective Changes , 2011 .

[66]  Hugh F. Durrant-Whyte,et al.  Sensor Models and Multisensor Integration , 1988, Int. J. Robotics Res..

[67]  Anthony J. Ries,et al.  EEG and Eye Tracking Signatures of Target Encoding during Structured Visual Search , 2017, Front. Hum. Neurosci..

[68]  Chen Liu,et al.  A Theoretical Model of Mental Workload in Pilots Based on Multiple Experimental Measurements , 2014, HCI.

[69]  Rosalind W. Picard,et al.  Automated Posture Analysis for Detecting Learner's Interest Level , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[70]  B. Cain A Review of the Mental Workload Literature , 2007 .

[71]  Hasan Ayaz,et al.  Optical brain monitoring for operator training and mental workload assessment , 2012, NeuroImage.

[72]  Louis-Philippe Morency,et al.  Multimodal Machine Learning: A Survey and Taxonomy , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Hussein A. Abbass,et al.  Towards a Code of Best Practice for Evaluating Air Traffic Control Interfaces , 2011 .

[74]  Yang Wang,et al.  Using galvanic skin response for cognitive load measurement in arithmetic and reading tasks , 2012, OZCHI.

[75]  F. T. Eggemeier,et al.  Recommendations for Mental Workload Measurement in a Test and Evaluation Environment , 1993 .

[76]  Yuan Qi,et al.  The Bayes Point Machine for computer-user frustration detection via pressuremouse , 2001, PUI '01.

[77]  Gahangir Hossain,et al.  When does an easy task become hard? A systematic review of human task-evoked pupillary dynamics versus cognitive efforts , 2016, Neural Computing and Applications.

[78]  Maarten A. Hogervorst,et al.  Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload , 2014, Front. Neurosci..

[79]  Peter A Hancock,et al.  State of science: mental workload in ergonomics , 2015, Ergonomics.

[80]  G. Borghini,et al.  Neuroscience and Biobehavioral Reviews , 2022 .

[81]  Julie A. Adams,et al.  A Survey of Workload Assessment Algorithms , 2018, IEEE Transactions on Human-Machine Systems.

[82]  Magdalena Borys,et al.  An analysis of EEG signal combined with pupillary response in the dynamics of human cognitive processing , 2016, 2016 9th International Conference on Human System Interactions (HSI).

[83]  J. Kettenring,et al.  Canonical Analysis of Several Sets of Variables , 2022 .

[84]  Jari Jussila,et al.  Application for pre-processing and visualization of electrodermal activity wearable data , 2017 .

[85]  Kyriaki Kalimeri,et al.  Identifying Urban Mobility Challenges for the Visually Impaired with Mobile Monitoring of Multimodal Biosignals , 2016, HCI.

[86]  James C. Christensen,et al.  The effects of day-to-day variability of physiological data on operator functional state classification , 2012, NeuroImage.

[87]  Yang Wang,et al.  Multimodal behavior and interaction as indicators of cognitive load , 2012, TIIS.

[88]  Xu Huang,et al.  Physiological fluctuations show frequency-specific networks in fNIRS signals during resting state , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[89]  Fang Chen,et al.  Automatic Cognitive Load Detection from Face, Physiology, Task Performance and Fusion During Affective Interference , 2014, Interact. Comput..

[90]  Daniel J. Barber,et al.  Augmenting Robot Behaviors Using Physiological Measures of Workload State , 2016, HCI.

[91]  Keng-Liang Ou,et al.  Toward a functional near-infrared spectroscopy-based monitoring of pain assessment for nonverbal patients , 2017, Journal of biomedical optics.

[92]  Andrzej Cichocki,et al.  Bimodal BCI Using Simultaneously NIRS and EEG , 2014, IEEE Transactions on Biomedical Engineering.

[93]  G. Salvendy,et al.  Review and reappraisal of modelling and predicting mental workload in single- and multi-task environments , 2000 .

[94]  Luca Longo,et al.  Human-Computer Interaction and Human Mental Workload: Assessing Cognitive Engagement in the World Wide Web , 2011, INTERACT.

[95]  Jing Fan,et al.  Multimodal Fusion for Cognitive Load Measurement in an Adaptive Virtual Reality Driving Task for Autism Intervention , 2015, HCI.