Automatically Generating Specification Properties From Task Models for the Formal Verification of Human–Automation Interaction

Human-automation interaction (HAI) is often a contributor to failures in complex systems. This is frequently due to system interactions that were not anticipated by designers and analysts. Model checking is a method of formal verification analysis that automatically proves whether or not a formal system model adheres to desirable specification properties. Task analytic models can be included in formal system models to allow HAI to be evaluated with model checking. However, previous work in this area has required analysts to manually formulate the properties to check. Such a practice can be prone to analyst error and oversight which can result in unexpected dangerous HAI conditions not being discovered. To address this, this paper presents a method for automatically generating specification properties from task models that enables analysts to use formal verification to check for system HAI problems they may not have anticipated. This paper describes the design and implementation of the method. An example (a pilot performing a before landing checklist) is presented to illustrate its utility. Limitations of this approach and future research directions are discussed.

[1]  Ellen J. Bass,et al.  Generating Erroneous Human Behavior From Strategic Knowledge in Task Models and Evaluating Its Impact on System Safety With Model Checking , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[2]  Ellen J. Bass,et al.  Formally verifying human–automation interaction as part of a system model: limitations and tradeoffs , 2010, Innovations in Systems and Software Engineering.

[3]  Natarajan Shankar,et al.  The SAL Language Manual , 2003 .

[4]  Michael D. Harrison,et al.  Systematic Analysis of Control Panel Interfaces Using Formal Tools , 2008, DSV-IS.

[5]  Ashish Tiwari,et al.  Sal 2 , 2004, CAV.

[6]  Matthew L. Bolton,et al.  Automatic validation and failure diagnosis of human-device interfaces using task analytic models and model checking , 2013, Comput. Math. Organ. Theory.

[7]  Robert E. Fields,et al.  Analysis of erroneous actions in the design of critical systems , 2001 .

[8]  Asaf Degani,et al.  Taming HAL: Designing Interfaces Beyond 2001 , 2004 .

[9]  Philippe A. Palanque,et al.  Formal modelling of incidents and accidents as a means for enriching training material for satellite control operations , 2008 .

[10]  Nadine B. Sarter,et al.  How in the World Did We Ever Get into That Mode? Mode Error and Awareness in Supervisory Control , 1995, Hum. Factors.

[11]  Matthew B. Dwyer,et al.  Analyzing interaction orderings with model checking , 2004 .

[12]  Ellen J. Bass,et al.  A Systematic Approach to Model Checking Human–Automation Interaction Using Task Analytic Models , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[13]  Philippe A. Palanque,et al.  A tool-supported design framework for safety critical interactive systems , 2003, Interact. Comput..

[14]  Constance L. Heitmeyer,et al.  On the Need for Practical Formal Methods , 1998, FTRTFT.

[15]  Ellen J. Bass,et al.  Using task analytic models to visualize model checker counterexamples , 2010, 2010 IEEE International Conference on Systems, Man and Cybernetics.

[16]  Gregory D. Abowd,et al.  A formal technique for automated dialogue development , 1995, Symposium on Designing Interactive Systems.

[17]  Inseok Hwang,et al.  Immediate observability of discrete event systems with application to user-interface design , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[18]  Bruno Dutertre,et al.  Timed Systems in SAL , 2004 .

[19]  Christine M. Mitchell,et al.  A Discrete Control Model of Operator Function: A Methodology for Information Display Design , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[20]  Ann Blandford,et al.  Verification-guided modelling of salience and cognitive load , 2008, Formal Aspects of Computing.

[21]  Jon Damon Reese,et al.  Analyzing Software Specifications for Mode Confusion Potential , 1998 .

[22]  Erik Hollnagel,et al.  The Phenotype of Erroneous Actions , 1993, Int. J. Man Mach. Stud..

[23]  Fabio Paternò,et al.  Automatic Support for Usability Evaluation , 1998, IEEE Trans. Software Eng..

[24]  David E. Kieras,et al.  Using GOMS for user interface design and evaluation: which technique? , 1996, TCHI.

[25]  Alexandre M. Bayen,et al.  Hybrid verification of an interface for an automatic landing , 2002, Proceedings of the 41st IEEE Conference on Decision and Control, 2002..

[26]  Ellen J. Bass,et al.  Generating phenotypical erroneous human behavior to evaluate human-automation interaction using model checking , 2012, Int. J. Hum. Comput. Stud..

[27]  Mickaël Baron,et al.  Formal and experimental validation approaches in HCI systems design based on a shared event B model , 2006, International Journal on Software Tools for Technology Transfer.

[28]  Valerie L. Shalin,et al.  Cognitive task analysis , 2000 .

[29]  Sandra Basnyat,et al.  Error Patterns: Systematic Investigation of Deviations in Task Models , 2006, TAMODIA.

[30]  Philippe A. Palanque,et al.  Validating interactive system design through the verification of formal task and system models , 1995, EHCI.

[31]  Via S. Maria.  Formal reasoning about dialogue properties with automatic support , 1997 .

[32]  Jeannette M. Wing.  A specifier's introduction to formal methods , 1990, Computer.

[33]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[34]  David A. Maluf,et al.  On Space Exploration And Human Error - A Paper on Reliability and Safety , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[35]  Michael Feary,et al.  Automatic Detection of Interaction Vulnerabilities in an Executable Specification , 2007, HCI.

[36]  Asaf Degani,et al.  HMI aspects of automotive climate control systems , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[37]  Ann Blandford,et al.  Formal Modelling of Cognitive Interpretation , 2006, DSV-IS.

[38]  Raja Parasuraman,et al.  Human-Automation Interaction , 2005 .

[39]  Ellen J. Bass,et al.  Evaluating Human-Human Communication Protocols with Miscommunication Generation and Model Checking , 2013, NASA Formal Methods.

[40]  Michael D. Harrison,et al.  Formally verifying interactive systems: A review , 1997, DSV-IS.

[41]  Fabio Paternò.  A Theory of User-interaction Objects , 1994, J. Vis. Lang. Comput..

[42]  Lance Sherry,et al.  Improving the aircraft cockpit user-interface: using rule-based expert system models , 2001 .

[43]  L. Kohn,et al.  To Err Is Human : Building a Safer Health System , 2007 .

[44]  Carl A. Gunter,et al.  Specifying and Analyzing Workflows for Automated Identification and Data Capture , 2009 .

[45]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[46]  Fabio Paternò,et al.  Integrating Model Checking and HCI Tools to Help Designers Verify User Interface Properties , 2000, DSV-IS.

[47]  Heinrich Hussmann,et al.  Model-Based Testing of Infotainment Systems on the Basis of a Graphical Human-Machine Interface , 2010, 2010 Second International Conference on Advances in System Testing and Validation Lifecycle.

[48]  Ellen J. Bass,et al.  Using Model Checking to Explore Checklist-Guided Pilot Behavior , 2012 .

[49]  Ellen J. Bass,et al.  Using Formal Verification to Evaluate Human-Automation Interaction: A Review , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[50]  E. Allen Emerson,et al.  Temporal and Modal Logic , 1991, Handbook of Theoretical Computer Science, Volume B: Formal Models and Sematics.

[51]  F. Paternò.  Model-based design of interactive applications , 2000, INTL.

[52]  Philippe A. Palanque,et al.  Formal socio-technical barrier modelling for safety-critical interactive systems design , 2007 .

[53]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[54]  James Reason,et al.  Human Error , 1990 .

[55]  Matthew L Bolton,et al.  A Method for the Formal Verification of Human-interactive Systems. , 2009, Proceedings of the Human Factors and Ergonomics Society ... Annual Meeting. Human Factors and Ergonomics Society. Annual Meeting.

[56]  Ellen J. Bass,et al.  Formal Modeling of Erroneous Human Behavior and its Implications for Model Checking , 2008 .

[57]  Asaf Degani,et al.  Formal Verification of Human-Automation Interaction , 2002, Hum. Factors.

[58]  Ann Blandford,et al.  An approach to formal verification of human–computer interaction , 2007, Formal Aspects of Computing.

[59]  Karen M. Feigh,et al.  Formal Modeling and Analysis for Interactive Hybrid Systems , 2011 .

[60]  Ellen J. Bass,et al.  Using task analytic behavior modeling, erroneous human behavior generation, and formal methods to evaluate the role of human-automation interaction in system failure , 2010 .

[61]  Christine M. Mitchell,et al.  Using the operator function model and OFMspert as the basis for an intelligent tutoring system: towards a tutor/aid paradigm for operators of supervisory control systems , 1995, IEEE Trans. Syst. Man Cybern..

[62]  Fabio Paternò,et al.  Preventing user errors by systematic analysis of deviations from the system task model , 2002, Int. J. Hum. Comput. Stud..

[63]  J. C. Higgins,et al.  Human Factors Considerations with Respect to Emerging Technology in Nuclear Power Plants , 2008 .

[64]  Barry Kirwan,et al.  A Guide To Task Analysis: The Task Analysis Working Group , 1992 .

[65]  Using Formal Methods to Predict Human Error and System Failures , 2010 .

[66]  Michael D. Harrison,et al.  Interaction engineering using the IVY tool , 2009, EICS '09.

[67]  Hassan Reza,et al.  A Model-Based Approach for Testing GUI Using Hierarchical Predicate Transition Nets , 2007, Fourth International Conference on Information Technology (ITNG'07).

[68]  Michael D. Harrison,et al.  Analysing interactive devices based on information resource constraints , 2014, Int. J. Hum. Comput. Stud..

[69]  Ellen J. Bass,et al.  Architecture and development environment of a knowledge-based monitor that facilitate incremental knowledge-base development , 2004, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[70]  Mickaël Baron,et al.  Formal Validation of HCI User Tasks , 2003, Software Engineering Research and Practice.

[71]  S.P. Miller,et al.  Mode confusion analysis of a flight guidance system using formal methods , 2003, Digital Avionics Systems Conference, 2003. DASC '03. The 22nd.

[72]  Abraham Silberschatz,et al.  Operating System Concepts , 1983 .

[73]  Karsten Loer,et al.  An integrated framework for the analysis of dependable interactive systems (IFADIS): Its tool support and evaluation , 2006, Automated Software Engineering.

[74]  Ann Blandford,et al.  Formal Modelling of Salience and Cognitive Load , 2008, Electron. Notes Theor. Comput. Sci..