Three Experiments Examining the Use of Electroencephalogram, Event-Related Potentials, and Heart-Rate Variability for Real- Time Human-Centered Adaptive Automation Design

Adaptive automation represents an advanced form of human-centered automation design. The approach to automation provides for real-time and model-based assessments of human-automation interaction, determines whether the human has entered into a hazardous state of awareness and then modulates the task environment to keep the operator in-the-loop , while maintaining an optimal state of task engagement and mental alertness. Because adaptive automation has not matured, numerous challenges remain, including what the criteria are, for determining when adaptive aiding and adaptive function allocation should take place. Human factors experts in the area have suggested a number of measures including the use of psychophysiology. This NASA Technical Paper reports on three experiments that examined the psychophysiological measures of event-related potentials, electroencephalogram, and heart-rate variability for real-time adaptive automation. The results of the experiments confirm the efficacy of these measures for use in both a developmental and operational role for adaptive automation design. The implications of these results and future directions for psychophysiology and human-centered automation design are discussed.

[1]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[2]  V. A. M. Hunt,et al.  The Future of Air Traffic Control , 1961, The Journal of the Royal Aeronautical Society.

[3]  Gavriel Salvendy,et al.  Handbook of Human Factors and Ergonomics , 2005 .

[4]  E Donchin,et al.  Event-related brain potentials and subjective probability in a learning task , 1980, Memory & cognition.

[5]  John R. Wilson,et al.  Mental Models: Theory and Application in Human Factors , 1989 .

[6]  Raja Parasuraman,et al.  EFFECTS OF MANUAL AND AUTOPILOT CONTROL ON MENTAL WORKLOAD AND VIGILANCE DURING SIMULATED GENERAL AVIATION FLIGHT , 1999 .

[7]  J. P. C. de Weerd,et al.  A posteriori time-varying filtering of averaged evoked potentials , 1981, Biological Cybernetics.

[8]  J. Beatty,et al.  Contrasting effects of response uncertainty on the task-evoked pupillary response and reaction time. , 1987, Psychophysiology.

[9]  Clare D. McGillem,et al.  Measurements of Signal Components in Single Visualy Evoked Brain Potentials , 1977, IEEE Transactions on Biomedical Engineering.

[10]  J. Beatty,et al.  The pupillary system. , 2000 .

[11]  Opmeer Ch,et al.  Towards an objective assessment of cockpit workload. I. Physiological variables during different flight phases. , 1973 .

[12]  H. McIlvaine Parsons,et al.  Automation and the Individual: Comprehensive and Comparative Views , 1985 .

[13]  M. Rugg The effects of semantic priming and work repetition on event-related potentials. , 1985, Psychophysiology.

[14]  K. G. Wilson Electrodermal lability and simple reaction time , 1987, Biological Psychology.

[15]  Frank E. Gomer,et al.  Physiological Monitoring and the Concept of Adaptive Systems , 1981 .

[16]  R. Davidson EEG measures of cerebral asymmetry: conceptual and methodological issues. , 1988, The International journal of neuroscience.

[17]  B. Fowler P300 as a Measure of Workload during a Simulated Aircraft Landing Task , 1994, Human factors.

[18]  Mark S. Sanders,et al.  Human factors in engineering and design, 7th ed. , 1993 .

[19]  J. Andreassi Psychophysiology: Human Behavior and Physiological Response , 1980 .

[20]  R. R. Mackie,et al.  Vigilance: Theory, Operational Performance, and Physiological Correlates , 1978 .

[21]  G. Wilson Applied use of cardiac and respiration measures: Practical considerations and precautions , 1992, Biological Psychology.

[22]  Arthur F. Kramer,et al.  Assessing the development of automatic processing: An application of dual-task and event-related brain potential methodologies , 1988, Biological Psychology.

[23]  N. Squires,et al.  Two varieties of long-latency positive waves evoked by unpredictable auditory stimuli in man. , 1975, Electroencephalography and clinical neurophysiology.

[24]  Sem-Jacobsen Cw,et al.  Brain/computer communication to reduce human error: a perspective. , 1981 .

[25]  Raja Parasuraman,et al.  Event-related brain potentials and human factors research. , 1990 .

[26]  W. Herrmann,et al.  COMSTAT rule for vigilance classification based on spontaneous EEG activity. , 1987, Neuropsychobiology.

[27]  H. Petsche Cognitive components in cerebral event-related potentials and selective attention. Progress in clinical neurophysiology, Vol. 6:J.E. Desmedt (Ed.) (Karger, Basel, 322 p., 95 fig., 17 tab., 1979, U.S. $ 70.75) , 1980 .

[28]  C D Wickens,et al.  Assessment of pilot performance and mental workload in rotary wing aircraft. , 1993, Ergonomics.

[29]  Christoph Braun,et al.  Auditory and somatosensory event-related potentials: I. Effects of attention , 1991 .

[30]  Mark H. Chignell,et al.  Mental workload dynamics in adaptive interface design , 1988, IEEE Trans. Syst. Man Cybern..

[31]  W. S. Peavler,et al.  Incentive effects and pupillary changes in association learning. , 1969, Journal of experimental psychology.

[32]  J. Beatty,et al.  The pupil and stimulus probability. , 1985, Psychophysiology.

[33]  L. Kaufman,et al.  Handbook of perception and human performance , 1986 .

[34]  R W Backs,et al.  Psychophysiological Measures of Workload during Continuous Manual Performance , 1994, Human factors.

[35]  James F. O'Hanlon,et al.  Concurrence of Electroencephalographic and Performance Changes During a Simulated Radar Watch and Some Implications for the Arousal Theory of Vigilance , 1977 .

[36]  J F O'Hanlon,et al.  Operant Control of Occipital Theta Rhythm Affects Performance in a Radar Monitoring Task , 1974, Science.

[37]  E. Lindholm,et al.  Autonomic activity and workload during learning of a simulated aircraft carrier landing task. , 1983, Aviation, space, and environmental medicine.

[38]  中村 道彦,et al.  Contingent Negative Variation〔邦文〕 (脳脊髄誘発電位の基礎と臨床 ) , 1979 .

[39]  E. Donchin,et al.  Performance of concurrent tasks: a psychophysiological analysis of the reciprocity of information-processing resources. , 1983, Science.

[40]  Daniel Gopher,et al.  Workload: An examination of the concept. , 1986 .

[41]  M. R. Houck,et al.  Event-related potentials elicited by automatic targets: a dual-task analysis. , 1985, Journal of experimental psychology. Human perception and performance.

[42]  P A Hancock,et al.  Pilot performance and preference for short cycles of automation in adaptive function allocation. , 1995, Applied ergonomics.

[43]  C D Wickens,et al.  An Analysis of the Processing Requirements of a Complex Perceptual-Motor Task , 1983, Human factors.

[44]  John G. Carlson,et al.  International perspectives on self-regulation and health , 1991 .

[45]  Kurt Offenloch,et al.  Computer Aided Physiological Assessment of the Functional State of Pilots during Simulated Flight , 1990 .

[46]  A W Gaillard,et al.  Comparing the concepts of mental load and stress. , 1993, Ergonomics.

[47]  津田 敏雄,et al.  Contingent negative variation , 1984 .

[48]  M. A. Biferno Mental workload measurement: Event-related potentials and ratings of workload and fatigue , 1985 .

[49]  F. Thomas Eggemeier,et al.  Workload assessment methodology. , 1986 .

[50]  J. Beatty,et al.  Pupillary dilations in movement preparation and execution. , 1985, Psychophysiology.

[51]  R Kakigi,et al.  [Event-related brain potentials]. , 1997, Nihon rinsho. Japanese journal of clinical medicine.

[52]  Lawrence J. Prinzel,et al.  A Biocybernetic System for Examining Psychophysiological Correlates for Adaptive Automation , 1997 .

[53]  Rapid Thermal Multiprocessor,et al.  Supervisory Control of a , 1993 .

[54]  K. G. Wilson,et al.  Electrodermal lability and visual information processing. , 1989, Psychophysiology.

[55]  W. S. Peavler,et al.  Effects of verbalization and incentive on the pupil response to mental activity. , 1968, Canadian journal of psychology.

[56]  Walter W. Surwillo,et al.  Psychophysiology for Clinical Psychologists , 1990 .

[57]  T. Åkerstedt,et al.  Sleepiness on the job: continuously measured EEG changes in train drivers. , 1987, Electroencephalography and clinical neurophysiology.

[58]  L C Walrath,et al.  Blink activity in a discrimination task as a function of stimulus modality and schedule of presentation. , 1985, Psychophysiology.

[59]  E. Donchin,et al.  COGNITIVE PSYCHOPHYSIOLOGY: THE ENDOGENOUS COMPONENTS OF THE ERP , 1978 .

[60]  D. Damos Multiple-task performance , 2020 .

[61]  Christopher D. Wickens,et al.  The Event Related Cortical Potential as an Index of Task Workload , 1977 .

[62]  M. Coles,et al.  Event-related brain potentials: An introduction. , 1995 .

[63]  P. Jorna Spectral analysis of heart rate and psychological state: A review of its validity as a workload index , 1992, Biological Psychology.

[64]  H. Schwilden,et al.  Closed-loop feedback control of propofol anaesthesia by quantitative EEG analysis in humans. , 1989, British journal of anaesthesia.

[65]  Lawrence J. Prinzel,et al.  A Bio-Cybernetic System for Adaptive Automation , 1995 .

[66]  E. John,et al.  Evoked-Potential Correlates of Stimulus Uncertainty , 1965, Science.

[67]  K. Kalyanaraman,et al.  The Electroencephalogram in , 1968 .

[68]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[69]  Frank E. Gomer,et al.  Electrocortical Activity and Operator Workload: A Comparison of Changes in the Electroencephalogram and in Event-Related Potentials , 1981 .

[70]  S Sutton,et al.  Quantitative evoked potential correlates of the probability of events. , 1970, Psychophysiology.

[71]  S. Hillyard,et al.  Electrical Signs of Selective Attention in the Human Brain , 1973, Science.

[72]  E C Beck,et al.  Electrophysiology and behavior. , 1975, Annual review of psychology.

[73]  E Donchin,et al.  A metric for thought: a comparison of P300 latency and reaction time. , 1981, Science.

[74]  G. McCarthy,et al.  Augmenting mental chronometry: the P300 as a measure of stimulus evaluation time. , 1977, Science.

[75]  Michael J. Kelly The Future of Air Traffic Control: Human Operators and Automation Edited by Christopher D. Wickens, Anne S. Mavor, Raja Parasuraman, & James P. McGee. 1998, 319 pages, $44.95 Washington, DC: National Academy Press ISBN 0-309-06412-0 , 1999 .

[76]  Norbert Wiener,et al.  Extrapolation, Interpolation, and Smoothing of Stationary Time Series , 1964 .

[77]  G Belenky,et al.  Effects of simulated high altitude exposure on long-latency event-related brain potentials and performance. , 1993, Aviation, space, and environmental medicine.

[78]  R. Parasuraman,et al.  Psychophysiology and adaptive automation , 1996, Biological Psychology.

[79]  Mustapha Mouloua,et al.  Effects of Adaptive Task Allocation on Monitoring of Automated Systems , 1996, Hum. Factors.

[80]  Marta Kutas,et al.  Event-Related Brain Potentials in the Study of Consciousness , 1983 .

[81]  E. Donchin,et al.  P300 and tracking difficulty: evidence for multiple resources in dual-task performance. , 1980, Psychophysiology.

[82]  K. Abbott,et al.  The interfaces between flightcrews and modern flight deck systems , 1996 .

[83]  B. Rouse William,et al.  Adaptive aiding for human/computer control , 1988 .

[84]  Nadine B. Sarter,et al.  Pilot Interaction With Cockpit Automation II: An Experimental Study of Pilots’ Model and Awareness of the Flight Management System , 1994 .

[85]  Rocky Ross,et al.  Mental models , 2004, SIGA.

[86]  W J Ray,et al.  EEG activity during cognitive processing: influence of attentional factors. , 1985, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[87]  Christopher D. Wickens,et al.  The Future of Air Traffic Control: Human Operators and Automation , 1998 .

[88]  J P Krol,et al.  Towards an objective assessment of cockpit workload. I. Physiological variables during different flight phases. , 1973, Aerospace medicine.

[89]  G. Schwartz,et al.  Consciousness and Self-Regulation , 1976 .

[90]  Lawrence J. Prinzel,et al.  Feedback Contingencies and Bio-Cybernetic Regulation of Operator Workload , 1995 .

[91]  J. Fahrenberg,et al.  Recording methods in applied environments , 2000 .

[92]  Thomas M. Longridge,et al.  Physiological Assessment of Aircraft Pilot Workload in Simulated Landing and Simulated Hostile Threat Environments , 1984 .

[93]  Raja Parasuraman,et al.  Operator versus computer control of adaptive automation , 1993 .

[94]  H. Jasper Report of the committee on methods of clinical examination in electroencephalography , 1958 .

[95]  M. R. Houck,et al.  The role of attentional resources in automatic detection , 1983, Cognitive Psychology.

[96]  David C. Nagel,et al.  Human factors in aviation , 1988 .

[97]  A. Kramer,et al.  Toward a Psychophysiological Assessment of Dynamic Changes in Mental Workload , 1994, Human factors.

[98]  E. Sirevaag,et al.  A Psychophysiological Assessment of Operator Workload During Simulated Flight Missions , 1987, Human factors.

[99]  R. C. Oldfield The assessment and analysis of handedness: the Edinburgh inventory. , 1971, Neuropsychologia.

[100]  C. Brunia,et al.  Anticipatory response-relevant muscle activity, CNV amplitude and simple reaction time. , 1985, Electroencephalography and clinical neurophysiology.

[101]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[102]  Kara D. Federmeier,et al.  Event-related brain potentials. , 1990 .

[103]  Barry H. Kantowitz,et al.  PILOT WORKLOAD AND FLIGHTDECK AUTOMATION. , 1996 .

[104]  K Nakajima,et al.  The electroencephalogram. , 1968, Developmental medicine and child neurology.

[105]  D. A. Spyker,et al.  Development of techniques for measuring pilot workload , 1971 .

[106]  D. Kahneman Attention and Effort , 1973 .

[107]  G. Romani,et al.  Auditory evoked magnetic fields and electric potentials , 1990 .

[108]  R Parasuraman,et al.  AUTOMATION AND HUMAN PERFORMANCE IN AVIATION. IN: PRINCIPLES AND PRACTICE OF AVIATION PSYCHOLOGY , 2003 .

[109]  D. Kahneman,et al.  Pupillary, heart rate, and skin resistance changes during a mental task. , 1969, Journal of experimental psychology.

[110]  J. M. Peter,et al.  The Efficacy of Psychophysiological Measures for Implementing Adaptive Technology , 2001 .

[111]  S. Hillyard,et al.  Cross-modal selective attention effects on retinal, myogenic, brainstem, and cerebral evoked potentials. , 1990, Psychophysiology.

[112]  Earl L. Wiener,et al.  Human factors of advanced technology (glass cockpit) transport aircraft , 1989 .

[113]  P. Ullsperger,et al.  The P300 component of the event-related brain potential and mental effort. , 1988, Ergonomics.

[114]  Mustapha Mouloua,et al.  Automation Technology and Human Performance: Current Research and Trends , 1999 .

[115]  P. Hancock Human Factors Psychology , 1987 .

[116]  Gavriel Salvendy,et al.  Handbook of human factors. , 1987 .

[117]  M. Scherg,et al.  A Source Analysis of the Late Human Auditory Evoked Potentials , 1989, Journal of Cognitive Neuroscience.

[118]  D. Lindsley,et al.  Attention, Vigilance, and Cortical Evoked-Potentials in Humans , 1964, Science.

[119]  E. Donchin Event-related Brain Potentials: A Tool in the Study of Human Information Processing , 1979 .

[120]  William B. Rouse,et al.  Human-Computer Interaction in Multitask Situations , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[121]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[122]  Judith M. Ford,et al.  Attention effects on auditory evoked potentials to infrequent events , 1976, Biological Psychology.

[123]  A. D. Fisk,et al.  The effects of practice and task structure on components of the event-related brain potential. , 1986, Psychophysiology.

[124]  R. Johnson A triarchic model of P300 amplitude. , 1986, Psychophysiology.

[125]  C D Wickens,et al.  The Event-Related Brain Potential as an Index of Display-Monitoring Workload , 1980, Human factors.

[126]  A. Kramer,et al.  Physiological metrics of mental workload: A review of recent progress , 1990, Multiple-task performance.

[127]  J Beatty,et al.  Response selection and initiation in speeded reactions: a pupillometric analysis. , 1983, Journal of experimental psychology. Human perception and performance.

[128]  Susan G. Hill,et al.  Traditional and raw task load index (TLX) correlations: Are paired comparisons necessary? In A , 1989 .

[129]  C W Sem-Jacobsen Brain/computer communication to reduce human error: a perspective. , 1981, Aviation, space, and environmental medicine.

[130]  A. Pope,et al.  Biocybernetic system evaluates indices of operator engagement in automated task , 1995, Biological Psychology.

[131]  John L. Andreassi,et al.  Psychophysiology: Human Behavior & Physiological Response , 2000 .

[132]  E R John,et al.  Information Delivery and the Sensory Evoked Potential , 1967, Science.

[133]  Richard W. Moss,et al.  2010: The Symbionic Cockpit , 1986, IEEE Aerospace and Electronic Systems Magazine.

[134]  R. C. Oldfield THE ASSESSMENT AND ANALYSIS OF HANDEDNESS , 1971 .

[135]  Richard L. Horst,et al.  An overview of current approaches and future challenges in physiological monitoring , 1988 .

[136]  C. Wickens Engineering psychology and human performance, 2nd ed. , 1992 .

[137]  W. Karwowski International encyclopedia of ergonomics and human factors , 2001 .

[138]  Earl A. Alluisi,et al.  Effects of Temporal Stressors on Vigilance and Information Processing , 1977 .

[139]  William C. Howell Human factors in the workplace , 1994 .

[140]  P. Hancock,et al.  Adaptive control in human-machine systems , 1987 .

[141]  D. Woods,et al.  Automation Surprises , 2001 .

[142]  E. Donchin Multivariate analysis of event-related potential data: A tutorial review , 1978 .

[143]  D. Strayer,et al.  Attentional Requirements of Automatic and Controlled Processing , 1990 .

[144]  Arthur F. Kramer,et al.  Real-time measurement of mental workload: A feasibility study , 1990 .

[145]  E Donchin,et al.  Resource reciprocity: an event-related brain potentials analysis. , 1989, Acta psychologica.

[146]  Bruce Christie,et al.  Psychophysiology and the electronic workplace , 1989 .

[147]  F. Yuan,et al.  SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) , 1999 .

[148]  Brandall Y. Suyenobu,et al.  Application of quantitative EEG analysis to workload assessment in an advanced aircraft simulator , 1993 .

[149]  M. Coles,et al.  Energetics and Human Information Processing , 1986 .

[150]  C Sekiguchi,et al.  Evaluation method of mental workload under flight conditions. , 1978, Aviation, space, and environmental medicine.

[151]  A. Lindqvist,et al.  Heart rate variability, cardiac mechanics, and subjectively evaluated stress during simulator flight. , 1983, Aviation, space, and environmental medicine.

[152]  J. R. Comstock MAT - MULTI-ATTRIBUTE TASK BATTERY FOR HUMAN OPERATOR WORKLOAD AND STRATEGIC BEHAVIOR RESEARCH , 1994 .

[153]  Graham Wilson,et al.  Psychophysiological assessment of workload in multi-task environments , 2020 .

[154]  M. Sterman,et al.  Concepts and applications of EEG analysis in aviation performance evaluation , 1995, Biological Psychology.

[155]  R. Ragot,et al.  Perceptual and motor space representation: an event-related potential study. , 1984, Psychophysiology.

[156]  R Parasuraman,et al.  Designing automation for human use: empirical studies and quantitative models , 2000, Ergonomics.

[157]  F. T. Eggemeier Properties of Workload Assessment Techniques , 1988 .

[158]  Lisanne Bainbridge,et al.  Ironies of automation , 1982, Autom..

[159]  Jeffrey B. Henriques,et al.  Asymmetrical brain electrical activity discriminates between psychometrically-matched verbal and spatial cognitive tasks. , 1990, Psychophysiology.

[160]  J. Beatty Task-evoked pupillary responses, processing load, and the structure of processing resources. , 1982, Psychological bulletin.

[161]  Donald A. Norman,et al.  Some observations on mental models , 1987 .

[162]  S. Geisser,et al.  On methods in the analysis of profile data , 1959 .

[163]  G. Robert J. Hockey,et al.  Level of Operator Control and Changes in Heart Rate Variability during Simulated Flight Maintenance , 1995, Hum. Factors.

[164]  William B. Rouse,et al.  Adaptive Aiding for Human-Computer Control: Experimental Studies of Dynamic Task Allocation. , 1986 .

[165]  J. C. Smith,et al.  Electroencephalographic Correlates of Pilot Performance: Simulation and In-Flight Studies , 1988 .

[166]  A F Kramer The interpretation of the component structure of event-related brain potentials: an analysis of expert judgments. , 1985, Psychophysiology.

[167]  M. Redondo,et al.  Decrements in heart rate variability during memory search. , 1992, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[168]  E. Donchin,et al.  Psychophysiology : systems, processes, and applications , 1987 .

[169]  D. Ruchkin,et al.  Emitted and evoked P300 potentials and variation in stimulus probability. , 1975, Psychophysiology.

[170]  G Vossel,et al.  Electrodermal habituation speed and visual monitoring performance. , 1984, Psychophysiology.

[171]  S. Farmer,et al.  The effects of physical and semantic incongruities on auditory event-related potentials. , 1984, Electroencephalography and clinical neurophysiology.

[172]  Richard W. Backs,et al.  THE USE OF AUTONOMIC COMPONENTS TO IMPROVE CARDIOVASCULAR ASSESSMENT OF MENTAL WORKLOAD IN FLIGHT SIMULATION , 1999 .

[173]  H. Shibasaki,et al.  Human auditory and somatosensory event-related potentials: effects of response condition and age. , 1987, Electroencephalography and clinical neurophysiology.

[174]  Mustapha Mouloua,et al.  Automation and Human Performance : Theory and Applications , 1996 .

[175]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[176]  T. Picton,et al.  The N1 wave of the human electric and magnetic response to sound: a review and an analysis of the component structure. , 1987, Psychophysiology.

[177]  Renwick E. Curry,et al.  Flight-deck automation: promises and problems , 1980 .

[178]  J. Veltman,et al.  Physiological workload reactions to increasing levels of task difficulty. , 1998, Ergonomics.

[179]  Raja Parasuraman,et al.  Automation- Induced "Complacency": Development of the Complacency-Potential Rating Scale , 1993 .

[180]  C D Wickens,et al.  Processing of stimulus properties: evidence for dual-task integrality. , 1985, Journal of experimental psychology. Human perception and performance.

[181]  F. Freeman,et al.  A Closed-Loop System for Examining Psychophysiological Measures for Adaptive Task Allocation , 2000, The International journal of aviation psychology.

[182]  David D. Woods,et al.  Decomposing Automation : Apparent Simplicity , Real Complexity , 2003 .

[183]  Raja Parasuraman,et al.  Performance Consequences of Automation-Induced 'Complacency' , 1993 .

[184]  J. Lubar Discourse on the development of EEG diagnostics and biofeedback for attention-deficit/hyperactivity disorders , 1991, Biofeedback and self-regulation.