When the Going Gets Tough: Exploring Agent-based Models of Human Performance under Demanding Conditions

Rianne van Lambalgen Air traffic control, naval warfare and air combat fighting are examples of domains where highly demanding tasks have to be executed. For an optimal performance on such tasks, humans have to process a great amount of information. This can be impaired due to limitations in human cognition. To aid humans in their task performance and to overcome the aforementioned limitations in human cognition, automated systems can be of help. On the one hand, systems can contribute support in case of performance degradation. On the other hand, automation can be used to provide training for people that have to perform well in demanding circumstances. Human-like agents that serve as opponents or team mates in training simulations need to be based on knowledge of human performance.

[1]  John R Anderson,et al.  An integrated theory of the mind. , 2004, Psychological review.

[2]  David B. Kaber,et al.  Adaptive Automation of a Dynamic Control Task Based on Secondary Task Workload Measurement , 1999 .

[3]  Ronald A. Rensink,et al.  Change blindness: past, present, and future , 2005, Trends in Cognitive Sciences.

[4]  Tibor Bosse,et al.  A Two-Level BDI-Agent Model for Theory of Mind and its Use in Social Manipulation , 2007 .

[5]  David B. Kaber,et al.  Situation awareness and workload in driving while using adaptive cruise control and a cell phone , 2005 .

[6]  Ronald S. Landis,et al.  The Effects of Cognitive Ability and Conscientiousness on Performance Over Time: A Censored Latent Growth Model , 2007 .

[7]  James Shanteau,et al.  Psychological characteristics of expert decision makers , 1987 .

[8]  Steven K. Feiner,et al.  Directing attention and influencing memory with visual saliency modulation , 2011, CHI.

[9]  Chris M. Herdman,et al.  The Role of Working Memory in Supporting Drivers’ Situation Awareness for Surrounding Traffic , 2010 .

[10]  Glenn F. Wilson,et al.  Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural Networks , 2003, Hum. Factors.

[11]  Robert Plomin,et al.  Genetics and general cognitive ability (g) , 2002, Trends in Cognitive Sciences.

[12]  Daniel M Wegner,et al.  Précis of The illusion of conscious will , 2004, Behavioral and Brain Sciences.

[13]  Mica R. Endsley,et al.  Design and Evaluation for Situation Awareness Enhancement , 1988 .

[14]  Margaret Wilson,et al.  Six views of embodied cognition , 2002, Psychonomic bulletin & review.

[15]  Tibor Bosse,et al.  Simulation and Formal Analysis of Visual Attention in Cognitive Systems , 2008, WAPCV.

[16]  Brian F. Gore,et al.  A Computational Implementation of a Human Attention Guiding Mechanism in MIDAS v5 , 2009, HCI.

[17]  J. B. Levitt,et al.  Contrast dependence of contextual effects in primate visual cortex , 1997, nature.

[18]  Tibor Bosse,et al.  A Language and Environment for Analysis of Dynamics by Simulation , 2005, Int. J. Artif. Intell. Tools.

[19]  Alice F. Healy,et al.  A cognitive modeling account of simultaneous learning and fatigue effects , 2011, Cognitive Systems Research.

[20]  Glenn F. Wilson,et al.  Performance Enhancement in an Uninhabited Air Vehicle Task Using Psychophysiologically Determined Adaptive Aiding , 2007, Hum. Factors.

[21]  Michel C. A. Klein,et al.  An ambient agent system assisting humans in complex tasks by analysis of a human's state and performance , 2013, Int. J. Intell. Inf. Database Syst..

[22]  Deborah L. McGuinness,et al.  An Intelligent Personal Assistant for Task and Time Management , 2007, AI Mag..

[23]  Christopher D. Wickens,et al.  Multiple resources and performance prediction , 2002 .

[24]  P. V. Maanen.  Adaptive Support for Human-Computer Teams : Exploring the Use of Cognitive Models of Trust and Attention , 2010 .

[25]  Wai-Tat Fu,et al.  Soft constraints in interactive behavior: the case of ignoring perfect knowledge in-the-world for imperfect knowledge in-the-head , 2004, Cogn. Sci..

[26]  P. Hancock,et al.  Stress, Workload, and Fatigue , 2009 .

[27]  Xing Xie,et al.  A visual attention model for adapting images on small displays , 2003, Multimedia Systems.

[28]  Laurent Itti,et al.  A Bayesian model for efficient visual search and recognition , 2010, Vision Research.

[29]  Christopher D. Wickens,et al.  Humans: Still Vital After All These Years of Automation , 2008, Hum. Factors.

[30]  Glenn F. Wilson,et al.  Operator Functional State Classification Using Multiple Psychophysiological Features in an Air Traffic Control Task , 2003, Hum. Factors.

[31]  Liz Sonenberg,et al.  Situation awareness in intelligent agents: foundations for a theory of proactive agent behavior , 2004 .

[32]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[33]  C. L. Rose,et al.  The role of the Big Five personality factors in vigilance performance and workload , 2002 .

[34]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[35]  Nadine B. Sarter,et al.  Good Vibrations: Tactile Feedback in Support of Attention Allocation and Human-Automation Coordination in Event-Driven Domains , 1999, Hum. Factors.

[36]  R. Engle,et al.  The role of prefrontal cortex in working-memory capacity, executive attention, and general fluid intelligence: An individual-differences perspective , 2002, Psychonomic bulletin & review.

[37]  Michael P. Georgeff,et al.  Commitment and Effectiveness of Situated Agents , 1991, IJCAI.

[38]  Tibor Bosse,et al.  Specification and Verification of Dynamics in Agent Models , 2009, Int. J. Cooperative Inf. Syst..

[39]  P. McOwan,et al.  A semi-automated approach to balancing of bottom-up salience for predicting change detection performance. , 2010, Journal of vision.

[40]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[41]  David W. Hill,et al.  The Critical Power Concept , 1993, Sports medicine.

[42]  Jan Treur,et al.  Adaptation and Validation of an Agent Model of Functional State and Performance for Individuals , 2009, PRIMA.

[43]  Levent Yilmaz,et al.  Validation and verification of social processes within agent-based computational organization models , 2006, Comput. Math. Organ. Theory.

[44]  P. Hancock,et al.  A Dynamic Model of Stress and Sustained Attention , 1989, Human factors.

[45]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[46]  Michel C. A. Klein,et al.  Design of an Optimal Automation System: Finding a Balance between a Human's Task Engagement and Exhaustion , 2011, IEA/AIE.

[47]  Stephen K. Reed,et al.  Differences in expert and novice situation awareness in naturalistic decision making , 1996, Int. J. Hum. Comput. Stud..

[48]  M. Posner,et al.  Orienting of Attention* , 1980, The Quarterly journal of experimental psychology.

[49]  M R Endsley,et al.  Level of automation effects on performance, situation awareness and workload in a dynamic control task. , 1999, Ergonomics.

[50]  A.W.K. Gaillard,et al.  Operator Functional State: The Assessment and Prediction of Human Performance Degradation in Complex Tasks , 2003 .

[51]  Timothy Coelli,et al.  An Introduction to Efficiency and Productivity Analysis , 1997 .

[52]  Mark Hoogendoorn,et al.  Modeling Situation Awareness in Human-Like Agents Using Mental Models , 2011, IJCAI.

[53]  S. Baron-Cohen.  Mindblindness: An Essay on Autism and Theory of Mind , 1997 .

[54]  D Dawson,et al.  A quantitative model of work-related fatigue: background and definition , 2001, Ergonomics.

[55]  L J Williams,et al.  Tunnel vision or general interference? Cognitive load and attentional bias are both important. , 1988, The American journal of psychology.

[56]  V. C. Padaki,et al.  Smart Vest: wearable multi-parameter remote physiological monitoring system. , 2008, Medical engineering & physics.

[57]  Becky L. Hooey,et al.  Human performance modeling in aviation , 2007 .

[58]  Michel C. A. Klein,et al.  An Agent Model for Analysis of Human Performance Quality , 2010, 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[59]  G. Dirkin.  Cognitive Tunneling: Use of Visual Information under Stress , 1983, Perceptual and motor skills.

[60]  Christopher D. Wickens,et al.  Supporting Situation Assessment through Attention Guidance and Diagnostic Aiding: The Benefits and Costs of Display Enhancement on Judgment Skill , 2012 .

[61]  Tibor Bosse,et al.  Simulation and formal analysis of visual attention , 2009, Web Intell. Agent Syst..

[62]  Dylan D. Schmorrow,et al.  Enhancing Mitigation in Augmented Cognition , 2007 .

[63]  J. A. Veltman,et al.  Differentiation of mental effort measures : consequences for adaptive automation , 2003 .

[64]  Christopher D. Wickens,et al.  Attention-Situation Awareness (A-SA) Model of Pilot Error , 2007 .

[65]  Yili Liu,et al.  Queuing Network Modeling of Driver Workload and Performance , 2006, IEEE Transactions on Intelligent Transportation Systems.

[66]  J. Theeuwes.  Endogenous and Exogenous Control of Visual Selection , 1994, Perception.

[67]  Mark Hoogendoorn,et al.  An Integrated Agent Model Addressing Situation Awareness and Functional State in Decision Making , 2011, PRIMA.

[68]  J. Veltman,et al.  Physiological workload reactions to increasing levels of task difficulty. , 1998, Ergonomics.

[69]  M. Endsley.  The role of situation awareness in naturalistic decision making , 1997 .

[70]  Tibor Bosse,et al.  Model-Based Reasoning Methods within an Ambient Intelligent Agent Model , 2007, AmI Workshops.

[71]  Derrick J. Parkhurst,et al.  Modeling the role of salience in the allocation of overt visual attention , 2002, Vision Research.

[72]  Randolph M. Jones,et al.  Automated Intelligent Pilots for Combat Flight Simulation , 1998, AI Mag..

[73]  Anand S. Rao,et al.  BDI Agents: From Theory to Practice , 1995, ICMAS.

[74]  Mark Hoogendoorn,et al.  Learning Belief Connections in a Model for Situation Awareness , 2011, PRIMA.

[75]  H. Aarts,et al.  Priming and authorship ascription: when nonconscious goals turn into conscious experiences of self-agency. , 2009, Journal of personality and social psychology.

[76]  Mark W. Scerbo,et al.  Effects of a Psychophysiological System for Adaptive Automation on Performance, Workload, and the Event-Related Potential P300 Component , 2003, Hum. Factors.

[77]  Kevin O'Brien,et al.  Human Behavior Models for Agents in Simulators and Games: Part I: Enabling Science with PMFserv , 2006, Presence: Teleoperators & Virtual Environments.

[78]  F. Freeman,et al.  A Closed-Loop System for Examining Psychophysiological Measures for Adaptive Task Allocation , 2000, The International journal of aviation psychology.

[79]  David B. Kaber,et al.  The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task , 2004 .

[80]  Raja Parasuraman,et al.  Adaptive Automation for Human-Robot Teaming in Future Command and Control Systems , 2007 .

[81]  P A Hancock,et al.  Influence of task demand characteristics on workload and performance. , 1995, The International journal of aviation psychology.

[82]  Ron Henderson,et al.  Electronic Performance Monitoring: A Laboratory Investigation of the Influence of Monitoring and Difficulty on Task Performance, Mood State, and Self-Reported Stress Levels , 2000 .

[83]  Tibor Bosse,et al.  Attention Manipulation for Naval Tactical Picture Compilation , 2009, 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology.

[84]  Gary Klein,et al.  The effect of acute stressors on decision making. , 1996 .

[85]  John K. Tsotsos,et al.  Attention in Cognitive Systems, 5th International Workshop on Attention in Cognitive Systems, WAPCV 2008, Fira, Santorini, Greece, May 12, 2008, Revised Selected Papers , 2009, WAPCV.

[86]  M. Endsley.  Situation Awareness In Aviation Systems , 1999 .

[87]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[88]  Lester Ingber,et al.  Simulated annealing: Practice versus theory , 1993 .

[89]  Daniel J. Garland,et al.  Situation Awareness Analysis and Measurement , 2009 .

[90]  Susana Rubio,et al.  Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA‐TLX, and Workload Profile Methods , 2004 .

[91]  G. Matthews,et al.  Task-induced fatigue states and simulated driving performance , 2002, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[92]  A. Pope,et al.  Biocybernetic system evaluates indices of operator engagement in automated task , 1995, Biological Psychology.

[93]  E. Salas,et al.  The science of training: a decade of progress. , 2001, Annual review of psychology.

[94]  Daniel Kahneman,et al.  Remarks on attention control , 1970 .

[95]  M. A. Recarte,et al.  Mental workload while driving: effects on visual search, discrimination, and decision making. , 2003, Journal of experimental psychology. Applied.

[96]  Giovanni Galfano,et al.  Color, form and luminance capture attention in visual search , 2000, Vision Research.

[97]  Cleotilde Gonzalez,et al.  Situation Awareness of Commanders: A Cognitive Model , 2004 .

[98]  P. Todd,et al.  Simple Heuristics That Make Us Smart , 1999 .

[99]  Dick de Waard,et al.  Adaptive Task Support Based on Dynamic Human State Estimation , 2010, 2010 Fourth IEEE International Conference on Self-Adaptive and Self-Organizing Systems Workshop.

[100]  Stephen J. Boies,et al.  Components of attention. , 1971 .

[101]  Tinghuai Ma,et al.  Context-aware implementation based on CBR for smart home , 2005, WiMob'2005), IEEE International Conference on Wireless And Mobile Computing, Networking And Communications, 2005..

[102]  Michel C. A. Klein,et al.  An Agent Memory Model Enabling Rational and Biased Reasoning , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[103]  Christopher D. Wickens,et al.  Identifying Black Swans in NextGen: Predicting Human Performance in Off-Nominal Conditions , 2009, Hum. Factors.

[104]  Thomas E. Nygren,et al.  The Subjective Workload Assessment Technique: A Scaling Procedure for Measuring Mental Workload , 1988 .

[105]  Tibor Bosse,et al.  Automated Visual Attention Manipulation , 2009, WAPCV.

[106]  Cleotilde Gonzalez,et al.  Task Workload and Cognitive Abilities in Dynamic Decision Making , 2005, Hum. Factors.

[107]  J. A. Veltman,et al.  How to Manage Cognitive Task Load During Supervision and Damage Control in an All-Electric Ship , 2004 .

[108]  Pattie Maes,et al.  Situated agents can have goals , 1990, Robotics Auton. Syst..

[109]  A W Gaillard,et al.  Comparing the concepts of mental load and stress. , 1993, Ergonomics.

[110]  Joseph Y. Halpern,et al.  Modeling Belief in Dynamic Systems, Part I: Foundations , 1997, Artif. Intell..

[111]  Fokie Cnossen,et al.  Strategic changes in task performance in simulated car driving as an adaptive response to task demands , 2000 .

[112]  Hongbin Wang,et al.  A Model of Spatio-Temporal Coding of Memory for Multidimensional Stimuli , 2019, Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society.

[113]  Jan Treur,et al.  Compositional Design and Reuse of a Generic Agent Model . , 2000 .

[114]  Patrick Brézillon,et al.  Lecture Notes in Artificial Intelligence , 1999 .

[115]  H. Aarts,et al.  Positive affect as implicit motivator: on the nonconscious operation of behavioral goals. , 2005, Journal of personality and social psychology.

[116]  R Parasuraman,et al.  Designing automation for human use: empirical studies and quantitative models , 2000, Ergonomics.

[117]  Dario D. Salvucci.  Modeling Driver Behavior in a Cognitive Architecture , 2006, Hum. Factors.

[118]  Syed Waqar Jaffry,et al.  Analysis and Validation of Models for Trust Dynamics , 2011 .

[119]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[120]  Mica R. Endsley,et al.  Theoretical Underpinnings of Situation Awareness, A Critical Review , 2000 .

[121]  Christopher D. Wickens,et al.  Situation Awareness and Workload in Aviation , 2002 .

[122]  Francis T. Durso,et al.  Situation Awareness: Understanding Dynamic Environments , 2008, Hum. Factors.

[123]  Yangchun Li,et al.  The Second Choice , 2013 .

[124]  Wayne D. Gray,et al.  Milliseconds Matter: an Introduction to Microstrategies and to Their Use in Describing and Predicting Interactive Behavior Milliseconds Matter: an Introduction to Microstrategies and to Their Use in Describing and Predicting Interactive Behavior , 2022 .

[125]  Christian Lebiere,et al.  Modeling Pilot Performance with an Integrated Task Network and Cognitive Architecture Approach , 2007 .

[126]  Mark Hoogendoorn,et al.  Relating Personality and Physiological Measurements to Task Performance Quality , 2009 .

[127]  Harold W. Sorenson,et al.  Parameter estimation: Principles and problems , 1980 .

[128]  Jadwiga Indulska,et al.  Personalising Context-Aware Applications , 2005, OTM Workshops.

[129]  S. Fairclough,et al.  The influence of performance feedback on goal-setting and mental effort regulation , 2009 .

[130]  Tibor Bosse,et al.  Design and Validation of a Model for a Human's Functional State and Performance , 2011, Int. J. Model. Simul. Sci. Comput..

[131]  J. Salgado.  The Five Factor Model of personality and job performance in the European Community. , 1997, The Journal of applied psychology.

[132]  G. Hockey,et al.  Effects of time-pressure on decision-making under uncertainty: changes in affective state and information processing strategy. , 2000, Acta psychologica.

[133]  Raja Parasuraman,et al.  Adaptive Automation for Human Supervision of Multiple Uninhabited Vehicles: Effects on Change Detection, Situation Awareness, and Mental Workload , 2009 .

[134]  Garrett Molholt,et al.  Human-Technology Interaction: Evaluations of Spontaneous Engaged Speech , 2012 .

[135]  R. Parasuraman,et al.  Psychophysiology and adaptive automation , 1996, Biological Psychology.

[136]  Niels Taatgen,et al.  ACT-R Models of Cognitive Control in the Abstract Decision Making Task , 2007 .

[137]  Mark Hoogendoorn,et al.  An Agent Model for Decision Making Based upon Experiences Applied in the Domain of Fighter Pilots , 2010, 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[138]  Paul T. Costa,et al.  The Five-Factor Model of Personality and Its Relevance to Personality Disorders , 1992 .

[139]  L. F. Barrett,et al.  Individual differences in working memory capacity and dual-process theories of the mind. , 2004, Psychological bulletin.

[140]  V. Verma,et al.  Real-time fault detection and situational awareness for rovers: report on the Mars technology program task , 2004, 2004 IEEE Aerospace Conference Proceedings (IEEE Cat. No.04TH8720).

[141]  Marci S. DeCaro,et al.  From poor performance to success under stress: working memory, strategy selection, and mathematical problem solving under pressure. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[142]  Jan Wielemaker,et al.  An Overview of the SWI-Prolog Programming Environment , 2003, WLPE.

[143]  Tibor Bosse,et al.  An Agent Model for a Human's Functional State and Performance , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[144]  Daniel G. Morrow,et al.  Expertise Differences in Attentional Strategies Related to Pilot Decision Making , 2008, Hum. Factors.

[145]  G. R. J. Hockey.  Compensatory control in the regulation of human performance under stress and high workload: A cognitive-energetical framework , 1997, Biological Psychology.

[146]  Stephen F. Smith,et al.  CMRadar: A Personal Assistant Agent for Calendar Management , 2004, AAAI.

[147]  Michael J. Frank,et al.  Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia , 2006, Neural Computation.

[148]  John A. Wise,et al.  Handbook of aviation human factors , 2009 .

[149]  Tibor Bosse,et al.  A system to support attention allocation: Development and application , 2012, Web Intell. Agent Syst..

[150]  Robert Plomin,et al.  Genetics and general cognitive ability , 1999, Nature.

[151]  Wayne D. Gray,et al.  The soft constraints hypothesis: a rational analysis approach to resource allocation for interactive behavior. , 2006, Psychological review.

[152]  Patrick Le Callet,et al.  A coherent computational approach to model bottom-up visual attention , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[153]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[154]  Raja Parasuraman,et al.  Effects of Imperfect Automation on Decision Making in a Simulated Command and Control Task , 2007, Hum. Factors.