GT-Cats: Tracking Operator Activities in Complex Systems

Human operators of complex dynamic systems can experience difficulties supervising advanced control automation. One remedy is to develop intelligent aiding systems that can provide operators with context-sensitive advice and reminders. The research reported herein proposes, implements, and evaluates a methodology for activity tracking, a form of intent inferencing that can supply the knowledge required for an intelligent aid by constructing and maintaining a representation of operator activities in real time. The methodology was implemented in the Georgia Tech Crew Activity Tracking System (GT-CATS), which predicts and interprets the actions performed by Boeing 757/767 pilots navigating using autopilot flight modes. This report first describes research on intent inferencing and complex modes of automation. It then provides a detailed description of the GT-CATS methodology, knowledge structures, and processing scheme. The results of an experimental evaluation using airline pilots are given. The results show that GT-CATS was effective in predicting and interpreting pilot actions in real time.

[1]  James Reason,et al.  Human Error , 1990 .

[2]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[3]  Robert S Dodd,et al.  A Review and Discussion of Flight Management System Incidents Reported to the Aviation Safety Reporting System , 1992 .

[4]  D. Norman.  Categorization of action slips. , 1981 .

[5]  Roger C. Schank,et al.  SCRIPTS, PLANS, GOALS, AND UNDERSTANDING , 1988 .

[6]  Joong Nam Kim,et al.  Agent-based aids to facilitate cockpit task management , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[7]  William J. Clancey,et al.  Knowledge-based tutoring: the GUIDON program , 1987 .

[8]  T. Govindaraj,et al.  Operator modeling in civil aviation , 1990, 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings.

[9]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[10]  Christine M. Mitchell,et al.  Use of Model-Based Qualitative Icons and Adaptive Windows in Workstations for Supervisory Control Systems , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[11]  R. E. Curry,et al.  Human factors of flight-deck automation: report on a NASA-industry workshop , 1983 .

[12]  David C. Nagel,et al.  Human factors in aviation , 1988 .

[13]  R. E. Curry.  The Introduction of New Cockpit Technology: A Human Factors Study , 1985 .

[14]  Christine M. Mitchell,et al.  Intent Inferencing with a Model-Based Operator’s Associate , 1990 .

[15]  Lisanne Bainbridge,et al.  Verbal reports as evidence of the process operator's knowledge , 1999, Int. J. Hum. Comput. Stud..

[16]  Formal Papers.  Pilot Interaction With Cockpit Automation 11: An Experimental Study of Pilots' Model and Awareness of the Flight Management System , 1994 .

[17]  Christine M. Mitchell,et al.  A methodology for understanding how operators select and use modes of automation to control complex dynamic systems , 1994, Proceedings of IEEE International Conference on Systems, Man and Cybernetics.

[18]  Erik Hollinagel.  Issues in knowledge-based decision support , 1987 .

[19]  René Amalberti,et al.  Cognitive Modelling of Fighter Aircraft Process Control: A Step Towards an Intelligent On-Board Assistance System , 1992, Int. J. Man Mach. Stud..

[20]  David C. Nagel,et al.  Pilots of the Future: Human or Computer? , 1985, Computer.

[21]  D A Norman,et al.  The 'problem' with automation: inappropriate feedback and interaction, not 'over-automation'. , 1990, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[22]  Delmar M. Fadden,et al.  Flight Deck Automation Today - Where Do We Go from Here? , 1987 .

[23]  R. John Hansman,et al.  Mode Awareness in Advanced Autoflight Systems , 1995 .

[24]  Abigail Sellen,et al.  The Prevention of Mode Errors Through Sensory Feedback , 1992, Hum. Comput. Interact..

[25]  F. Hawkins.  Human factors in aviation. , 1979, Journal of psychosomatic research.

[26]  Christine M. Mitchell,et al.  OFMspert: inference of operator intentions in supervisory control using a blackboard architecture , 1988, IEEE Trans. Syst. Man Cybern..

[27]  Lucy Suchman.  Plans and situated actions: the problem of human-machine communication , 1987 .

[28]  H. Simon,et al.  Models of Man. , 1957 .

[29]  Ronald M. Baecker,et al.  Readings in human-computer interaction : a multidisciplinary approach , 1988 .

[30]  Alex Kirlik,et al.  Mode Usage in Automated Cockpits: Some Initial Observations , 1995 .

[31]  David Woods,et al.  Behind human error : cognitive systems, computers, and hindsight : state-of-the-art report , 1994 .

[32]  Andrew P. Sage,et al.  Handbook of Systems Engineering and Management , 2011 .

[33]  Donald A. Norman,et al.  User Centered System Design , 1986 .

[34]  Etienne Wenger,et al.  Artificial Intelligence and Tutoring Systems: Computational and Cognitive Approaches to the Communication of Knowledge , 1987 .

[35]  David D. Woods,et al.  Commentary: Cognitive Engineering in Complex and Dynamic Worlds , 1988, Int. J. Man Mach. Stud..

[36]  Kay Inckle,et al.  Strong and Silent , 2014 .

[37]  Kevin B. Bennett,et al.  Human Interaction with an "Intelligent" Machine , 1987, Int. J. Man Mach. Stud..

[38]  Christine M. Mitchell,et al.  Using the operator function model and OFMspert as the basis for an intelligent tutoring system: towards a tutor/aid paradigm for operators of supervisory control systems , 1995, IEEE Trans. Syst. Man Cybern..

[39]  Christine M. Mitchell,et al.  A methodology for the design of interactive monitoring interfaces , 1994, Proceedings of IEEE International Conference on Systems, Man and Cybernetics.

[40]  Guy A. Boy.  Operator Assistant Systems , 1987, Int. J. Man Mach. Stud..

[41]  H. Nii.  Blackboard Systems , 1998 .

[42]  A. Kirlik.  Requirements for psychological models to support design: Towards ecological task analysis , 1991 .

[43]  S Vandebuerie,et al.  OUT OF THE LOOP , 2004 .

[44]  Judith H. Lind,et al.  Agent-based pilot-vehicle interfaces: concept and prototype , 1992, IEEE Trans. Syst. Man Cybern..

[45]  Jens Rasmussen,et al.  Information Processing and Human-Machine Interaction , 1986 .

[46]  Thomas B. Sheridan,et al.  Telerobotics, Automation, and Human Supervisory Control , 2003 .

[47]  Christine M. Mitchell,et al.  Human-computer cooperative problem solving: theory, design, and evaluation of an intelligent associate system , 1995, IEEE Trans. Syst. Man Cybern..

[48]  Eric Harslem,et al.  Designing the STAR User Interface , 1987, ECICS.

[49]  Sandra H. Rouse,et al.  Design and Evaluation of an Onboard Computer-Based Information System for Aircraft , 1982, IEEE Transactions on Systems, Man, and Cybernetics.

[50]  Kim J. Vicente,et al.  Ecological interface design: theoretical foundations , 1992, IEEE Trans. Syst. Man Cybern..

[51]  David D. Woods,et al.  Cognitive Technologies: The Design of Joint Human-Machine Cognitive Systems , 1986, AI Mag..

[52]  Jrg Nievergelt,et al.  Sites, modes, and trails: Telling the user of an interactive system where he is, what he can do, and how to get to places (excerpt) , 1987 .

[53]  William B. Rouse,et al.  An Architecture for Intelligent Interfaces: Outline of an Approach to Supporting Operators of Complex Systems , 1987, Hum. Comput. Interact..

[54]  Christine M. Mitchell,et al.  A methodology for human-machine systems research: knowledge engineering, modeling, and simulation , 1995, IEEE Trans. Syst. Man Cybern..

[55]  Jens Rasmussen,et al.  Decision support in supervisory control of high-risk industrial systems , 1987, Autom..

[56]  G. Perlman.  Making the right choices with menus , 1987 .

[57]  Sheila B. Banks,et al.  Pilot's Associate: a cooperative, knowledge-based system application , 1991, IEEE Expert.

[58]  S. Greenberg,et al.  The Psychology of Everyday Things , 2012 .

[59]  Erik Hollnagel,et al.  Mapping Cognitive Demands in Complex Problem-Solving Worlds , 1987, Int. J. Man Mach. Stud..

[60]  Nadine B. Sarter,et al.  Mode Error in Supervisory Control of Automated Systems , 1992 .

[61]  Andrew F. Monk,et al.  Mode Errors: A User-Centered Analysis and Some Preventative Measures Using Keying-Contingent Sound , 1986, Int. J. Man Mach. Stud..

[62]  T. Landauer,et al.  Handbook of Human-Computer Interaction , 1997 .

[63]  Christine M. Mitchell,et al.  Addressing the trained novice/expert performance gap in complex dynamic systems: a case-based intelligent tutoring system , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[64]  Christine M. Mitchell,et al.  Effects of integrated flight path and terrain displays on controlled flight into terrain , 1993, Proceedings of IEEE Systems Man and Cybernetics Conference - SMC.

[65]  Renwick E. Curry,et al.  Flight-deck automation: promises and problems , 1980 .

[66]  N. D. Geddes.  Understanding human operators' intentions in complex systems , 1990 .

[67]  Jens Rasmussen,et al.  Cognitive Systems Engineering , 2022 .

[68]  William B. Rouse,et al.  Conceptual design of a human error tolerant interface for complex engineering systems , 1987, Autom..

[69]  C.M. Mitchell,et al.  ALLY: an operator's associate for cooperative supervisory control systems , 1993, IEEE Trans. Syst. Man Cybern..

[70]  E. M. Roth,et al.  Chapter 1 – Cognitive Systems Engineering , 1988 .

[71]  Nadine B. Sarter,et al.  Pilot Interaction With Cockpit Automation II: An Experimental Study of Pilots’ Model and Awareness of the Flight Management System , 1994 .

[72]  Donald A. Norman,et al.  Designing for error , 1987 .

[73]  Christine M. Mitchell,et al.  Operator Modeling: Conceptual and Methodological Distinctions , 1987 .