Developing neurophysiological metrics for the assessment of mental workload and the functional state of the brain

OF DOCTORAL DISSERTATION AALTO UNIVERSITY SCHOOL OF SCIENCE AND TECHNOLOGY P.O. BOX 11000, FI-00076 AALTO http://www.aalto.fi Author Anu Holm Name of the dissertation DEVELOPING NEUROPHYSIOLOGICAL METRICS FOR THE ASSESSMENT OF MENTAL WORKLOAD AND THE FUNCTIONAL STATE OF THE BRAIN Manuscript submitted 15.12.2009 Manuscript revised 09.04.2010 Date of the defence 12.05.2010 Monograph Article dissertation (summary + original articles) Faculty Faculty of Information and Natural Sciences Department Department of Biomedical Engineering and Computational Science Field of research Neuroergonomics Opponent(s) Docent Minna Huotilainen, PhD Supervisor Professor Risto Ilmoniemi Instructor Docent Kiti Müller, MD Abstract Modern working environments are often information intensive and work performance requires acting on multiple tasks simultaneously, i.e., multitasking. Also, irregular and prolonged work schedules, shift work and night work are typical in many work sectors. This causes both acute and chronic sleep loss, which results in performance impairment, such as increased reaction times, memory difficulties, cognitive slowing, and lapses of attention. Long lasting sleep loss and sustained overloading increase the risk of human errors and may cause work related stress and even occupational burn-out.Modern working environments are often information intensive and work performance requires acting on multiple tasks simultaneously, i.e., multitasking. Also, irregular and prolonged work schedules, shift work and night work are typical in many work sectors. This causes both acute and chronic sleep loss, which results in performance impairment, such as increased reaction times, memory difficulties, cognitive slowing, and lapses of attention. Long lasting sleep loss and sustained overloading increase the risk of human errors and may cause work related stress and even occupational burn-out. According to the Finnish Occupational Safety and Health Act (738/2002, Työturvallisuuslaki), Section 25 Avoiding and reducing workloads, an employer should assess the workload the employee is exposed to. Despite the fact that this important issue is enacted in the law, the objective measures to assess the workload are lacking. This Thesis reviews neurophysiologic methods for assessment of cognitive workload and sleep loss. Then it describes experimental studies where the feasibility of conventional event related potential (ERP) and electroencephalography (EEG) methods were tested both in assessment of internal state of participants during challenging task performance after sleep debt and in diagnostic of work-related central nervous system disorder. After that, methodological improvements both on ERPs and EEG metrics are shown: ERPs were analysed with a single-trial method, and EEG methodology was developed for estimation of both internal (caused by sleep loss) and external (caused by task demands) load. The methods were tested in healthy controls. The most promising metric to study overall brain load, including both cognitive workload as well as sleep loss, is suggested to be theta Fz / alpha Pz -ratio. It increases both with growing cognitive workload level and time spent awake, being sensitive also to sleep loss. This metric is possible to measure both in laboratory and in the field conditions. Measurements may be carried out even during real work tasks, at least in professions where most work is done in office-like environments. As the ratio increases with cognitive brain load similar to the heartbeat with increasing physical load, the ratio was named "brainbeat".

[1]  E. Donchin,et al.  Is the P300 component a manifestation of context updating? , 1988, Behavioral and Brain Sciences.

[2]  James L. Szalma,et al.  Stress and Neuroergonomics , 2006, Neuroergonomics.

[3]  P300 sequence effects differ between children and adults for auditory stimuli. , 1999, Psychophysiology.

[4]  J. Hohnsbein,et al.  Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. , 1991, Electroencephalography and clinical neurophysiology.

[5]  Timothy F. Elsmore,et al.  SYNWORK1: A PC-based tool for assessment of performance in a simulated work environment , 1994 .

[6]  T. Sejnowski,et al.  Removing electroencephalographic artifacts by blind source separation. , 2000, Psychophysiology.

[7]  C D Wickens,et al.  The Event-Related Brain Potential as an Index of Display-Monitoring Workload , 1980, Human factors.

[8]  A. Kramer,et al.  Physiological metrics of mental workload: A review of recent progress , 1990, Multiple-task performance.

[9]  Kilseop Ryu,et al.  Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic , 2005 .

[10]  S. Hillyard,et al.  Electrical Signs of Selective Attention in the Human Brain , 1973, Science.

[11]  Louise Venables,et al.  The influence of task demand and learning on the psychophysiological response. , 2005, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[12]  A. Pope,et al.  Biocybernetic system evaluates indices of operator engagement in automated task , 1995, Biological Psychology.

[13]  A J Nash,et al.  P300 and allocation of attention in dual-tasks. , 1996, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[14]  P Ullsperger,et al.  Event-related potential correlates of individual differences in working memory capacity. , 1999, Psychophysiology.

[15]  F. Thomas Eggemeier,et al.  Workload assessment methodology. , 1986 .

[16]  Glenn F. Wilson,et al.  Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural Networks , 2003, Hum. Factors.

[17]  Robert T. Knight,et al.  Neurobiology: sleep on it. , 2004 .

[18]  E Pellouchoud,et al.  Neurophysiological signals of working memory in normal aging. , 2001, Brain research. Cognitive brain research.

[19]  Timothy A. Salthouse,et al.  Determinants of adult age differences on synthetic work performance. , 1996 .

[20]  Michael E. Smith,et al.  Monitoring Working Memory Load during Computer-Based Tasks with EEG Pattern Recognition Methods , 1998, Hum. Factors.

[21]  Gregory Belenky,et al.  Individual differences in vulnerability to sleep loss in the work environment. , 2009, Industrial health.

[22]  Rex L. Cannon,et al.  EEG Biofeedback as a Treatment for Substance Use Disorders: Review, Rating of Efficacy, and Recommendations for Further Research , 2008, Applied psychophysiology and biofeedback.

[23]  F. Freeman,et al.  Evaluation of an adaptive automation system using three EEG indices with a visual tracking task , 1999, Biological Psychology.

[24]  T. Porkka-Heiskanen,et al.  Adenosine, Energy Metabolism, and Sleep , 2003, TheScientificWorldJournal.

[25]  Nadine B. Sarter,et al.  Neuroergonomics: Opportunities and challenges of merging cognitive neuroscience with cognitive ergonomics , 2003 .

[26]  R. McCarley,et al.  Adenosine: a mediator of the sleep-inducing effects of prolonged wakefulness. , 1997, Science.

[27]  W. Klimesch,et al.  Relevance of EEG alpha and theta oscillations during task switching , 2006, Experimental Brain Research.

[28]  Raja Parasuraman,et al.  Neuroergonomics: Research and practice , 2003 .

[29]  Robert W. Proctor,et al.  An empirical evaluation of the SYNWORK1 multiple-task work environment , 1998 .

[30]  P Maquet,et al.  The Role of Sleep in Learning and Memory , 2001, Science.

[31]  A. Yli-Hankala,et al.  Description of the Entropy™ algorithm as applied in the Datex‐Ohmeda S/5™ Entropy Module , 2004, Acta anaesthesiologica Scandinavica.

[32]  Barrett S. Caldwell,et al.  Human Performance and Control of Multiple Systems , 1996, Hum. Factors.

[33]  Glenn F. Wilson,et al.  Operator Functional State Classification Using Multiple Psychophysiological Features in an Air Traffic Control Task , 2003, Hum. Factors.

[34]  J. L. Kenemans,et al.  The effects of sleep loss on task performance and the electroencephalogram in young and elderly subjects , 1997, Biological Psychology.

[35]  Michael E. Smith,et al.  Monitoring Task Loading with Multivariate EEG Measures during Complex Forms of Human-Computer Interaction , 2001, Hum. Factors.

[36]  Anders M. Fjell,et al.  P300 and Neuropsychological Tests as Measures of Aging: Scalp Topography and Cognitive Changes , 2004, Brain Topography.

[37]  Joseph R. Madsen,et al.  Human theta oscillations exhibit task dependence during virtual maze navigation , 1999, Nature.

[38]  Fuvro Yevloe,et al.  Frontal midline theta rhythm and eyeblinking activity during a VDT task and a video game : useful tools for psychophysiology in ergonomics , 2012 .

[39]  K. Müller,et al.  Sleep restriction for the duration of a work week impairs multitasking performance , 2010, Journal of sleep research.

[40]  S. Slobounov,et al.  Neurophysiological and behavioral indices of time pressure effects on visuomotor task performance. , 2000, Brain research. Cognitive brain research.

[41]  L. McEvoy,et al.  The impact of moderate sleep loss on neurophysiologic signals during working-memory task performance. , 2002, Sleep.

[42]  Isao Hashimoto,et al.  Sequential information processing during a mental arithmetic is reflected in the time course of event-related brain potentials , 2000, Clinical Neurophysiology.

[43]  U. Erdmann,et al.  Auditory probe sensitivity to mental workload changes - an event-related potential study. , 2001, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[44]  E. Golob,et al.  Effects of stimulus sequence on event-related potentials and reaction time during target detection in Alzheimer's disease , 2000, Clinical Neurophysiology.

[45]  A. Kok Age-related changes in involuntary and voluntary attention as reflected in components of the event-related potential (ERP) , 2000, Biological Psychology.

[46]  H. Jasper,et al.  The ten-twenty electrode system of the International Federation. The International Federation of Clinical Neurophysiology. , 1999, Electroencephalography and clinical neurophysiology. Supplement.

[47]  A. Jefferson Offutt,et al.  An Empirical Evaluation , 1994 .

[48]  E Donchin,et al.  A new method for off-line removal of ocular artifact. , 1983, Electroencephalography and clinical neurophysiology.

[49]  C. Gonsalvez,et al.  Target-to-target interval versus probability effects on P300 in one- and two-tone tasks. , 2003, Psychophysiology.

[50]  Piotr Jaskowski,et al.  Evidence for an Integrative Role of P3b in Linking Reaction to Perception , 2005 .

[51]  L. Kaufman,et al.  Handbook of perception and human performance , 1986 .

[52]  Daniel Gopher,et al.  Workload: An examination of the concept. , 1986 .

[53]  N. Squires,et al.  The effect of stimulus sequence on the waveform of the cortical event-related potential. , 1976, Science.

[54]  J. Kaipio,et al.  Subspace regularization method for the single-trial estimation of evoked potentials , 1999, IEEE Transactions on Biomedical Engineering.

[55]  G. McCarthy,et al.  Augmenting mental chronometry: the P300 as a measure of stimulus evaluation time. , 1977, Science.

[56]  D Friedman,et al.  Age-related changes in scalp topography to novel and target stimuli. , 1993, Psychophysiology.

[57]  F. Díaz,et al.  Effect of the relevance and position of the target stimuli on P300 and reaction time. , 2001, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[58]  G. Wilson,et al.  Cognitive task classification based upon topographic EEG data , 1995, Biological Psychology.

[59]  J. Polich,et al.  EEG and ERP assessment of normal aging. , 1997, Electroencephalography and clinical neurophysiology.

[60]  S. Fairclough,et al.  Prediction of subjective states from psychophysiology: A multivariate approach , 2006, Biological Psychology.

[61]  J. Cohen,et al.  On the number of trials needed for P300. , 1997, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[62]  Raja Parasuraman,et al.  Monitoring an Automated System for a Single Failure: Vigilance and Task Complexity Effects , 1996, Hum. Factors.

[63]  Robert J Barry,et al.  Single-trial event-related potentials to significant stimuli. , 2009, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[64]  T. Finkenzeller,et al.  Frontal midline theta in the pre-shot phase of rifle shooting: Differences between experts and novices , 2008, Neuropsychologia.

[65]  W. Klimesch,et al.  Theta synchronization and alpha desynchronization in a memory task. , 1997, Psychophysiology.

[66]  E. Donchin Multivariate analysis of event-related potential data: A tutorial review , 1978 .

[67]  Carolyne R Swain,et al.  Electrophysiological, behavioral, and subjective indexes of workload when performing multiple tasks: manipulations of task difficulty and training. , 1999, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[68]  K. Cote,et al.  Waking quantitative electroencephalogram and auditory event-related potentials following experimentally induced sleep fragmentation. , 2003, Sleep.

[69]  E Donchin,et al.  Resource reciprocity: an event-related brain potentials analysis. , 1989, Acta psychologica.

[70]  Christopher D. Wickens,et al.  Multiple Resources and Mental Workload , 2008, Hum. Factors.

[71]  J. Koninck,et al.  Total sleep deprivation and novelty processing: implications for frontal lobe functioning , 2005, Clinical Neurophysiology.

[72]  C. Woody Characterization of an adaptive filter for the analysis of variable latency neuroelectric signals , 1967, Medical and biological engineering.

[73]  Michael E. Smith,et al.  Neurophysiological measures of cognitive workload during human-computer interaction , 2003 .

[74]  C D Wickens,et al.  Processing of stimulus properties: evidence for dual-task integrality. , 1985, Journal of experimental psychology. Human perception and performance.

[75]  L. M. Ward,et al.  Synchronous neural oscillations and cognitive processes , 2003, Trends in Cognitive Sciences.

[76]  L. Kaufman,et al.  Handbook of Perception and Human Performance. Volume 2. Cognitive Processes and Performance , 1994 .

[77]  G. Hockey Changes in operator efficiency as a function of environmental stress, fatigue, and circadian rhythms. , 1986 .

[78]  P. Maquet Sleep on it! , 2000, Nature Neuroscience.

[79]  G. R. J. Hockey Compensatory control in the regulation of human performance under stress and high workload: A cognitive-energetical framework , 1997, Biological Psychology.

[80]  Clare D. McGillem,et al.  Measurements of Signal Components in Single Visualy Evoked Brain Potentials , 1977, IEEE Transactions on Biomedical Engineering.

[81]  Christopher D. Wickens,et al.  Multiple resources and performance prediction , 2002 .

[82]  F. Yamada Frontal midline theta rhythm and eyeblinking activity during a VDT task and a video game: useful tools for psychophysiology in ergonomics. , 1998, Ergonomics.

[83]  G. Knapp,et al.  The impact of solvent mixtures on neurobehavioral performance: conclusions from epidemiological data. , 2008, Neurotoxicology.

[84]  M. Sterman,et al.  Concepts and applications of EEG analysis in aviation performance evaluation , 1995, Biological Psychology.

[85]  J L Kenemans,et al.  Event-related potentials and secondary task performance during simulated driving. , 2008, Accident; analysis and prevention.

[86]  G Bryan Young Continuous EEG monitoring in the ICU: challenges and opportunities. , 2009, The Canadian journal of neurological sciences. Le journal canadien des sciences neurologiques.

[87]  E. Donchin,et al.  P300 and tracking difficulty: evidence for multiple resources in dual-task performance. , 1980, Psychophysiology.

[88]  Barry Fowler,et al.  The differential effects of Sternberg short- and long-term memory scanning on the late Nd and P300 in a dual-task paradigm. , 2004, Brain research. Cognitive brain research.

[89]  M. Härmä New work times are here--are we ready? , 1998, Scandinavian journal of work, environment & health.

[90]  C D Wickens,et al.  The dynamics of P300 during dual-task performance. , 1980, Progress in brain research.

[91]  K. Müller,et al.  The effects of sleep debt and monotonous work on sleepiness and performance during a 12‐h dayshift , 2004, Journal of sleep research.

[92]  Mark W. Scerbo,et al.  A brain-based system for adaptive automation , 2003 .

[93]  M. A. Postma,et al.  Fz theta divided by Pz alpha as an index of task load during a PC-based air traffic control simulation , 2005 .