Human performance modeling and its uncertainty factors affecting decision making: a survery

This paper introduces the background and connotation of human performance modeling (HPM), HPM models, and the application of artificial intelligence algorithms in HPM. It deeply analyzes the connotation and uncertainty of each model and finally puts forward its military application. The aim is to provide relevant researchers in the field with an in-depth understanding of domain knowledge and related uncertainties and to indicate future research directions. The first part is a general overview of human factors engineering, where the definition, origin, research field, importance, and general problems of HPM are elaborated. The composition of the man–machine system and its corresponding relationship with the observe–orient–decide–act loop are described. The second part reviews the models of perception, cognition, understanding, and decision making. Among them, models of cognition consist of visual search, visual sampling, mental workload, and goals, operators, methods, and selection rules; models of action consist of Hick–Hyman law, Fitts’s law, and manual control theory. The third part is a review of the source and importance of the integrated models and focuses on the principles, composition, and successful application cases of the three models, namely SAINT, IMPRINT, and ACT-R. The fourth part is a review of the application of the algorithms and models in the fields of artificial intelligence, deep learning, and data mining in analyzing multivariate datasets in HPM. In addition, future HPM military applications are presented.

[1]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[2]  K. R. Hammond.  Probabilistic functioning and the clinical method. , 1955, Psychological review.

[3]  Yili Liu,et al.  Investigation of Driver Performance With Night Vision and Pedestrian Detection Systems—Part I: Empirical Study on Visual Clutter and Glance Behavior , 2010, IEEE Transactions on Intelligent Transportation Systems.

[4]  Carl R. Eklund.  The Antarctic Skua , 1964 .

[5]  Allen Newell,et al.  The keystroke-level model for user performance time with interactive systems , 1980, CACM.

[6]  M. Back,et al.  Accuracy of judging personality , 2016 .

[7]  Yili Liu,et al.  Queueing Network-Model Human Processor (QN-MHP): A computational architecture for multitask performance in human-machine systems , 2006, TCHI.

[8]  Jeffrey T. Hansberger,et al.  Human Performance Modeling for Operational Command, Control and Communication , 2005 .

[9]  G. Srivastava,et al.  ICA-based procedures for removing ballistocardiogram artifacts from EEG data acquired in the MRI scanner , 2005, NeuroImage.

[10]  K. LandauerT.,et al.  Selection from alphabetic and numeric menu trees using a touch screen , 1985 .

[11]  Sheldon Baron,et al.  Perspectives on Human Performance Modelling , 1982 .

[12]  Hailin Ai,et al.  How to Optimize the Input Efficiency of Keyboard Buttons in Large Smartphone? A Comparison of Curved Keyboard and Keyboard Area Size , 2019, HCI.

[13]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[14]  L. Kaufman,et al.  Handbook of Perception and Human Performance. Volume 2. Cognitive Processes and Performance , 1994 .

[15]  I. Janis,et al.  Decision Making: A Psychological Analysis of Conflict, Choice, and Commitment , 1977 .

[16]  Waldemar Karwowski,et al.  A Review of Human Factors Challenges of Complex Adaptive Systems , 2012, Hum. Factors.

[17]  Ann M. Bisantz,et al.  Measuring the Fit between Human Judgments and Automated Alerting Algorithms: A Study of Collision Detection , 2003, Hum. Factors.

[18]  David E. Kieras,et al.  Using GOMS for user interface design and evaluation: which technique? , 1996, TCHI.

[19]  Stuart K. Card,et al.  Evaluation of mouse, rate-controlled isometric joystick, step keys, and text keys, for text selection on a CRT , 1987 .

[20]  T. K. Landauer,et al.  Selection from alphabetic and numeric menu trees using a touch screen: breadth, depth, and width , 1985, CHI '85.

[21]  Allan Fong,et al.  A Comparison of Artificial Neural Networks, Logistic Regressions, and Classification Trees for Modeling Mental Workload in Real-Time , 2010 .

[22]  D L Fisher,et al.  Minimizing the Time to Search Visual Displays: The Role of Highlighting , 1989, Human factors.

[23]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[24]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[25]  Wayne D. Gray,et al.  Milliseconds Matter: an Introduction to Microstrategies and to Their Use in Describing and Predicting Interactive Behavior Milliseconds Matter: an Introduction to Microstrategies and to Their Use in Describing and Predicting Interactive Behavior , 2022 .

[26]  Craig R Lawton,et al.  Human performance modeling for system of systems analytics :soldier fatigue. , 2005 .

[27]  Gary Witus,et al.  Computational Modeling of Foveal Target Detection , 2003, Hum. Factors.

[28]  John W. Senders,et al.  Visual Sampling Processes , 1984 .

[29]  Randy Goebel,et al.  Computational intelligence - a logical approach , 1998 .

[30]  Brian F Gore.  Human Performance Cognitive-Behavioral Modeling: A Benefit for Occupational Safety , 2002, International journal of occupational safety and ergonomics : JOSE.

[31]  Yili Liu,et al.  QN-ACTR Modeling of Multitask Performance of Dynamic and Complex Cognitive Tasks , 2012 .

[32]  John R Anderson,et al.  An integrated theory of the mind. , 2004, Psychological review.

[33]  David Kellmeyer,et al.  An Application of Human Performance Modeling to the Evaluation of Advanced User Interface Features , 2000 .

[34]  Christopher D. Wickens,et al.  When Users Want What's not Best for Them , 1995 .

[35]  Michael D. Byrne,et al.  A History and Primer of Human Performance Modeling , 2009 .

[36]  Becky L. Hooey,et al.  Human performance modeling in aviation , 2007 .

[37]  William J Horrey,et al.  Modeling drivers' visual attention allocation while interacting with in-vehicle technologies. , 2006, Journal of experimental psychology. Applied.

[38]  Michael S. Landy,et al.  Signal detection theory , 2009 .

[39]  Charneta Samms,et al.  Trade Study: A Two- Versus Three-Soldier Crew for the Mounted Combat System (MCS) and Other Future Combat System Platforms , 2003 .

[40]  Christopher D. Wickens,et al.  Using Meta-Analyses Results and Data Gathering to Support Human Performance Model Development , 2013 .

[41]  David E. Kieras,et al.  The GOMS family of user interface analysis techniques: comparison and contrast , 1996, TCHI.

[42]  Jeff Sauro.  Estimating Productivity: Composite Operators for Keystroke Level Modeling , 2009, HCI.

[43]  B. S. Everitt,et al.  Comprar The Cambridge Dictionary of Statistics | B. S. Everitt | 9780521766999 | Cambridge University Press , 2010 .

[44]  W. Ziemba,et al.  Growth-optimal investments and numeraire portfolios under transactions costs , 2013 .

[45]  Laurel Allender,et al.  Manpower, Personnel, and Training Integration Methods and Tools , 2005 .

[46]  Terence Soule,et al.  Assessing Mental Workload from Skin Conductance and Pupillometry using Wavelets and Genetic Programming , 2010 .

[47]  I. Scott MacKenzie,et al.  A Fitts' Law Evaluation of Hands-Free and Hands-On Input on a Laptop Computer , 2019, HCI.

[48]  Christopher D. Wickens,et al.  An introduction to human factors engineering , 1997 .

[49]  L. Kaufman,et al.  Handbook of perception and human performance , 1986 .

[50]  Yili Liu,et al.  Queueing network-adaptive control of thought rational (QN-ACTR): An integrated cognitive architecture for modelling complex cognitive and multi-task performance , 2013 .

[51]  Michael E. Atwood,et al.  Project Ernestine: Validating a GOMS Analysis for Predicting and Explaining Real-World Task Performance , 1993, Hum. Comput. Interact..

[52]  Antti Oulasvirta,et al.  Modelling Error Rates in Temporal Pointing , 2016, CHI.

[53]  W. Karwowski.  International encyclopedia of ergonomics and human factors , 2001 .

[54]  Christopher D. Wickens,et al.  Attentional Models of Multitask Pilot Performance Using Advanced Display Technology , 2003, Hum. Factors.

[55]  John A. Swets,et al.  Signal Detection Theory and ROC Analysis in Psychology and Diagnostics: Collected Papers , 1996 .

[56]  A. Tversky.  Elimination by aspects: A theory of choice. , 1972 .

[57]  Yke Bauke Eisma,et al.  Visual Sampling Processes Revisited: Replicating and Extending Senders (1983) Using Modern Eye-Tracking Equipment , 2018, IEEE Transactions on Human-Machine Systems.

[58]  R. Hogarth,et al.  Determinants of Linear Judgment: A Meta-Analysis of Lens Model Studies , 2007, Psychological bulletin.

[59]  Anthony D. Andre,et al.  Taxiway Navigation and Situation Awareness (T-NASA) System: Problem, Design Philosophy, and Description of an Integrated Display Suite for Low-Visibility Airport Surface Operations , 1996 .

[60]  M. Loeb,et al.  The Psychology of Vigilance , 1982 .

[61]  John W. Senders,et al.  The Human Operator as a Monitor and Controller of Multidegree of Freedom Systems , 1964 .

[62]  C. Wickens,et al.  Situation Awareness, Mental Workload, and Trust in Automation: Viable, Empirically Supported Cognitive Engineering Constructs , 2008 .

[63]  Anupam Agrawal,et al.  Vision based hand gesture recognition for human computer interaction: a survey , 2012, Artificial Intelligence Review.

[64]  Gary Klein,et al.  Reflections on applications of naturalistic decision making , 2015 .

[65]  Robert W. Proctor,et al.  Human factors in simple and complex systems , 1993 .

[66]  R. Hyman.  Stimulus information as a determinant of reaction time. , 1953, Journal of experimental psychology.

[67]  J. Swets,et al.  A decision-making theory of visual detection. , 1954, Psychological review.

[68]  Arthur I. Siegel,et al.  Man-machine simulation models : psychosocial and performance interaction , 1969 .

[69]  C. Lebiere,et al.  The Atomic Components of Thought , 1998 .

[70]  J. Wolfe,et al.  Five factors that guide attention in visual search , 2017, Nature Human Behaviour.

[71]  A. Kramer,et al.  Physiological metrics of mental workload: A review of recent progress , 1990, Multiple-task performance.

[72]  R. Dawes,et al.  Linear models in decision making. , 1974 .

[73]  Tahira Reid,et al.  Dynamic Data Driven Approach for Modeling Human Error , 2015, ICCS.

[74]  Judi E. See,et al.  Meta-analysis of the sensitivity decrement in vigilance. , 1995 .

[75]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[76]  John W. Payne,et al.  The adaptive decision maker: Name index , 1993 .

[77]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[78]  Özgür Simsek,et al.  SECONDARY-TASK MEASURES OF DRIVER WORKLOAD , 2001 .

[79]  Christopher D. Wickens,et al.  Using Empirical Research and Computational Modeling to Predict Operator Response to Unexpected Events , 2014 .

[80]  Changxu Wu,et al.  Integrating Behavior Modeling with Data Mining to Improve Human Error Prediction in Numerical Data Entry , 2014 .

[81]  Michael E. Atwood,et al.  The precis of Project Ernestine or an overview of a validation of GOMS , 1992, CHI.

[82]  Mei Rong,et al.  Human Factor Quantitative Analysis Based on OHFAM and Bayesian Network , 2014, HCI.

[83]  Séverine Deneulin,et al.  An introduction to the human development and capability approach : freedom and agency , 2009 .

[84]  E. Brunswik,et al.  The Conceptual Framework of Psychology , 1954 .

[85]  J. Neumann,et al.  Theory of games and economic behavior , 1945, 100 Years of Math Milestones.

[86]  J. Jonides.  Voluntary versus automatic control over the mind's eye's movement , 1981 .

[87]  Anthony D. Andre,et al.  A HUMAN-CENTERED METHODOLOGY FOR THE DESIGN, EVALUATION, AND INTEGRATION OF COCKPIT DISPLAYS , 2002 .

[88]  Joe Tekli,et al.  Evaluating Fitts' law on vibrating touch-screen to improve visual data accessibility for blind users , 2018, Int. J. Hum. Comput. Stud..

[89]  Melanie J. Martin,et al.  Cross-Level Effects Between Neurophysiology and Communication During Team Training , 2016, Hum. Factors.

[90]  Gary Klein,et al.  Sources of Power: How People Make Decisions , 2017 .

[91]  Arthur D. Fisk,et al.  Modeling and analysis of a dynamic judgment task using a lens model approach , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[92]  Raymond R. Bond,et al.  A Web-based Human Computer Interaction Audit Tool to Support Collaborative Cognitive Ergonomics Within Interaction Design , 2016, ECCE.

[93]  James L. Szalma,et al.  Training for Vigilance: Using Predictive Power to Evaluate Feedback Effectiveness , 2006, Hum. Factors.

[94]  W. E. Hick.  Quarterly Journal of Experimental Psychology , 1948, Nature.

[95]  Christopher D. Wickens,et al.  Attention-Situation Awareness (A-SA) Model of Pilot Error , 2007 .

[96]  Yili Liu,et al.  Mental Workload Modeling in an Integrated Cognitive Architecture , 2011 .

[97]  Carl Gutwin,et al.  A predictive model of menu performance , 2007, CHI.

[98]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[99]  Clark Borst,et al.  Touchscreens for Aircraft Navigation Tasks: Comparing Accuracy and Throughput of Three Flight Deck Interfaces Using Fitts’ Law , 2019, Human factors.

[100]  G. Robert J. Hockey,et al.  Level of Operator Control and Changes in Heart Rate Variability during Simulated Flight Maintenance , 1995, Hum. Factors.

[101]  David E. Kieras,et al.  A validation of the GOMS model methodology in the development of a specialized, commercial software application , 1994, CHI '94.

[102]  Eric J. Johnson,et al.  The adaptive decision maker , 1993 .

[103]  John M. Flach,et al.  Control Theory for Humans: Quantitative Approaches To Modeling Performance , 2002 .

[104]  A. Tversky,et al.  Prospect Theory : An Analysis of Decision under Risk Author ( s ) : , 2007 .

[105]  Yili Liu,et al.  Integrating Queueing Network and ACT-R Cognitive Architectures , 2011 .