Handbook of Human Performance Measures and Crew Requirements for Flightdeck Research.

Abstract : The Federal Aviation Administration (FAA) Technical Center envisions that their studies will require standard measure of pilot/crew performance. Therefore, the FAA commissioned the Crew System Ergonomics Information Analysis Center (CSERIAC) to (1) identify state-of-the-art pilot(crew performance measures in selected areas of interest, (2) provide guidance material to allow the FAA Technical Center to determine appropriate measures for a given study classification, and (3) provide guidelines on pilot subject characteristics used in their studies. Adhering to accepted standards will allow performance data to he translated between FAA studies and generalized across other government and industry partners. Three areas of human performance that have achieved the most attention in the literature are: workload, situational awareness, and vigilance. An extensive literature search was conducted on each of these areas and leading experts in the human performance research industry were consulted. The tools and techniques used to measure each of these three areas are investigated. Guidelines are provided to assist the human factors practitioner in choosing the most appropriate performance measure for a given study classification (e.g., part-task, full mission, end-to-end). A set of criteria and guidelines on pilot subject characteristics, such as, number of subjects, experience level required, and the we of different airline flightcrews, is also provided.

[1]  Michael A. Vidulich,et al.  Training and Subjective Workload in a Category Search Task , 1986 .

[2]  S G Hart,et al.  Measuring moment-to-moment pilot workload using synchronous presentations of secondary tasks in a motion-based trainer. , 1989, Aviation, space, and environmental medicine.

[3]  Arthur N. Beare,et al.  The Effects of Supervisor Experience and the Presence of a Shift Technical Advisor on the Performance of Two-Man Crews in a Nuclear Power Plant Simulator , 1984 .

[4]  F. T. Eggemeier,et al.  The Effect of Delayed Report on Subjective Ratings of Mental Workload , 1983 .

[5]  G. Borg.  Subjective aspects of physical and mental load. , 1978, Ergonomics.

[6]  Charles Graham,et al.  Effects of Pyridostigmine on Psychomotor and Visual Performance , 1984 .

[7]  F. Thomas Eggemeier,et al.  Subjective Workload Assessment in a Memory Update Task , 1982 .

[8]  Joel S. Warm,et al.  Vigilance and Workload , 1991 .

[9]  Gary B. Reid,et al.  A Comparative Evaluation of Two Subjective Workload Measures: The Subjective Workload Assessment Technique and the Modified Cooper Harper Scale. , 1986 .

[10]  Stephen P. Boyd.  Assessing the Validity of SWAT as a Workload Measurement Instrument , 1983 .

[11]  Martin L. Fracker.  Attention Allocation in Situation Awareness , 1989 .

[12]  Pamela S. Tsang,et al.  Assessing Subjective Workload Assessment: A Comparison of SWAT and the NASA-Bipolar Methods , 1985 .

[13]  Philip J. Masline.  A comparison of the sensitivity of interval scale psychometric techniques in the assessment of subjective mental workload , 1986 .

[14]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[15]  J. Lebacqz,et al.  A Flight Investigation of Control, Display, and Guidance Requirements for Decelerating Descending VTOL Instrument Transitions Using the X-22A Variable Stability Aircraft. Volume 2. Background Information and Supporting Data , 1975 .

[16]  L. Kaufman,et al.  Handbook of perception and human performance , 1986 .

[17]  Christopher D. Wickens,et al.  The Structure of Attentional Resources , 1980 .

[18]  S. G. Hart,et al.  Assessing the Subjective Workload of Directional Orientation Tasks , 1984 .

[19]  John G. Casali,et al.  A Validated Rating Scale for Global Mental Workload Measurement Applications , 1983 .

[20]  F. Thomas Eggemeier,et al.  The Effect of Intervening Task Performance on Subjective Workload Ratings , 1984 .

[21]  Scott S Potter,et al.  Subjective Workload Assessment Technique (SWAT): A User's Guide , 1989 .

[22]  M. A. Vidulich,et al.  The Use of Judgment Matrices in Subjective Workload Assessment: The Subjective Workload Dominance (SWORD) Technique , 1989 .

[23]  Reid Gb,et al.  The subjective perception of workload in low-time private pilots: a preliminary study. , 1987 .

[24]  E. J. Lovesey.  Contemporary ergonomics 1990 : proceedings of the Ergonomics Society's 1990 Annual Conference, Leeds, England, 3-6 April 1990 : ‘ergonomics-setting standards for the 90'sk' , 1990 .

[25]  D. Gopher,et al.  On the Psychophysics of Workload: Why Bother with Subjective Measures? , 1984 .

[26]  W W Wierwille,et al.  On the measurement of pilot perceptual workload: a comparison of assessment techniques addressing sensitivity and intrusion issues. , 1984, Ergonomics.

[27]  Thomas E Nygren,et al.  Conjoint Measurement and Conjoint Scaling: A Users Guide. , 1982 .

[28]  M. A. Biferno.  Mental workload measurement: Event-related potentials and ratings of workload and fatigue , 1985 .

[29]  F. Thomas Eggemeier,et al.  Workload assessment methodology. , 1986 .

[30]  Alan H Roscoe.  In-Flight Assessment of Workload Using Pilot Ratings and Heart Rate , 1987 .

[31]  Susan G. Hill,et al.  Subjective Workload Assessment during 48 Continuous Hours of LOS-F-H Operations , 1989 .

[32]  F. T. Eggemeier.  Properties of Workload Assessment Techniques , 1988 .

[33]  Loran A. Haworth,et al.  An Investigation of Single-Piloted Advanced Cockpit and Control Configurations for Nap-of-the-Earth Helicopter Combat Mission Tasks , 1986 .

[34]  Jacqueline T. Rehmann,et al.  The Relationship between Effort Rating and Performance in a Critical Tracking Task. , 1982 .

[35]  F. Thomas Eggemeier,et al.  Application of Conjoint Measurement to Workload Scale Development , 1981 .

[36]  George P. Boucek,et al.  Assessment of Crew Workload Measurement Methods, Techniques and Procedures. Volume 1. Process, Methods and Results , 1989 .

[37]  W. Wierwille,et al.  A Comparison of Rating Scale, Secondary-Task, Physiological, and Primary-Task Workload Estimation Techniques in a Simulated Flight Task Emphasizing Communications Load , 1983, Human factors.

[38]  Mica R. Endsley,et al.  Predictive Utility of an Objective Measure of Situation Awareness , 1990 .

[39]  William F. Moroney,et al.  Subjects in Human Factors: Just who are They? , 1992 .

[40]  P. S. Tsang,et al.  Evaluation of 2 cognitive abilities tests in a dual-task environment , 1986 .

[41]  F. D. Newell,et al.  A study of relationships between aircraft system performance and pilot ratings , 1970 .

[42]  E. Sirevaag,et al.  A Psychophysiological Assessment of Operator Workload During Simulated Flight Missions , 1987, Human factors.

[43]  William F. Storm,et al.  FB-111A Aircrew Use of Temazepam during Surge Operations, , 1987 .

[44]  Thomas E Nygren.  Investigation of an Error Theory for Conjoint Measurement Methodology. , 1983 .

[45]  Mica R. Endsley,et al.  Situation awareness global assessment technique (SAGAT) , 1988, Proceedings of the IEEE 1988 National Aerospace and Electronics Conference.

[46]  P. S. Tsang,et al.  Techniques of subjective workload assessment - A comparison of two methodologies , 1985 .

[47]  Marion P. Kibbe.  Information Transfer from Intelligent EW Displays , 1988 .

[48]  Vernol Battiste,et al.  Inflight evaluation of pilot workload measures for rotorcraft research , 1987 .

[49]  F. T. Eggemeier.  Workload Metrics for System Evaluatiion , 1983 .

[50]  Barry H. Kantowitz,et al.  Measuring Pilot Workload in a Moving-base Simulator. Part 2: Building Levels of Workload , 1984 .

[51]  Jerome C. Notestine,et al.  Subjective Workload Assessment and Effect of Delayed Ratings in a Probability Monitoring Task , 1984 .

[52]  Terence S. Abbott,et al.  Comparison of Workload Measures on Computer—Generated Primary Flight Displays , 1987 .

[53]  Arthur F. Kramer,et al.  Advanced Workload Assessment Techniques for Engineering Flight Simulation , 1991 .

[54]  W. W. Wierwille,et al.  Comparative evaluation of twenty pilot workload assessment measure using a psychomotor task in a moving base aircraft simulator , 1983 .

[55]  Walter W. Wierwille,et al.  Comparison of Five Mental Workload Assessment Procedures in a Moving-Base Driving Simulator , 1979 .

[56]  Mica R. Endsley,et al.  Design and Evaluation for Situation Awareness Enhancement , 1988 .

[57]  S. Metalis,et al.  Assessment of Pilot Situational Awareness: Measurement via Simulation , 1993 .

[58]  Thomas E. Nygren,et al.  Psychometric Properties of Subjective Workload Measurement Techniques: Implications for Their Use in the Assessment of Perceived Mental Workload , 1991 .

[59]  F. T. Eggemeier,et al.  Recommendations for Mental Workload Measurement in a Test and Evaluation Environment , 1993 .

[60]  Carlton Gamer.  Colorado Springs, Colorado , 1973 .

[61]  James D Wolf,et al.  Crew Workload Assessment. Development of a Measure of Operator Workload. , 1978 .

[62]  W W Johnson,et al.  Cognitive demands in automation. , 1989, Aviation, space, and environmental medicine.

[63]  Lawrence L Ames,et al.  Revision and Verification of a Seven-Point Workload Estimate Scale , 1993 .

[64]  George E. Cooper,et al.  Handling qualities and pilot evaluation , 1986 .

[65]  William L. Derrick,et al.  Examination of Workload Measures with Subjective Task Clusters , 1983 .

[66]  Pamela S. Tsang,et al.  Absolute Magnitude Estimation and Relative Judgement Approaches to Subjective Workload Assessment , 1987 .

[67]  Harold S. Blackman,et al.  Insights into Pilot Situation Awareness Using Verbal Protocol Analysis , 1991 .

[68]  Susan G. Hill,et al.  Generic Workload Ratings of a Mobile Air Defense System (LOS-F-H) , 1989 .

[69]  F. V. Schick,et al.  The Use of Subjective Workload Assessment Technique in a Complex Flight Task , 1987 .

[70]  W. W. Wietwille,et al.  Decision tree rating scales for workload estimation: Theme and variations , 1984 .

[71]  Leslie J. Peters,et al.  Tank Crew Performance: Effects of Speech Intelligibility on Target Acquisition and Subjective Workload Assessment , 1989 .

[72]  Thomas E. Nygren,et al.  The Subjective Workload Assessment Technique: A Scaling Procedure for Measuring Mental Workload , 1988 .

[73]  Sandra G. Hart,et al.  POPCORN: a Supervisory Control Simulation for Workload and Performance Research , 1984 .

[74]  Tsang Ps,et al.  Cognitive demands in automation. , 1989 .

[75]  R. G. Pearson,et al.  The Development and Validation of a Checklist for Measuring Subjective Fatigue , 1956 .

[76]  Pamela S. Tsang,et al.  Automation - Changes in cognitive demands and mental workload , 1987 .

[77]  Robert P. Bateman,et al.  Benefits of Using Objective and Subjective Workload Measures , 1984 .

[78]  Susan G. Hill,et al.  Workload Assessment of a Remotely Piloted Vehicle (RPV) System , 1988 .

[79]  S G Hart,et al.  Measuring pilot workload in a motion base trainer. A comparison of four techniques. , 1986, Applied ergonomics.

[80]  W W Wierwille,et al.  Evaluation of decision-tree rating scales for mental workload estimation. , 1986, Ergonomics.

[81]  Martin L. Fracker.  A Theory of Situation Assessment: Implications for Measuring Situation Awareness , 1988 .

[82]  Nadine B. Sarter,et al.  Pilot Interaction With Cockpit Automation II: An Experimental Study of Pilots’ Model and Awareness of the Flight Management System , 1994 .

[83]  Joel S. Warm,et al.  The role of choice in vigilance performance , 1992 .

[84]  P. G. A. M. Jorna,et al.  Operator Workload as a Limiting Factor in Complex Systems , 1991 .

[85]  David Woods,et al.  Situation Awareness: A Critical But Ill-Defined Phenomenon , 1991 .

[86]  Don E. Malzahn,et al.  Techniques of Subjective Assessment: A Comparison of the SWAT and Modified Cooper-Harper Scales , 1988 .

[87]  J. R. Comstock,et al.  An Initial Test of a Normative Figure of Merit for the Quality of Overall Task Performance , 1990 .

[88]  G B Reid,et al.  The subjective perception of workload in low-time private pilots: a preliminary study. , 1987, Aviation, space, and environmental medicine.

[89]  Earl S. Stein.  The Measurement of Pilot Performance: A Master-Journeyman Approach , 1984 .

[90]  Mansour Rahimi,et al.  Evaluation of 16 Measures of Mental Workload using a Simulated Flight Task Emphasizing Mediational Activity , 1985 .

[91]  David A. Sawin,et al.  Vigilance: It's Boring, It's Difficult, and I Can't do Anything about it , 1992 .

[92]  Michael A. Vidulich,et al.  A Dissociation of Objective and Subjective Workload Measures in Assessing the Impact of Speech Controls in Advanced Helicopters , 1988 .

[93]  R. M. Taylor,et al.  Workload or Situational Awareness?: TLX vs. SART for Aerospace Systems Design Evaluation , 1991 .

[94]  F. Thomas Eggemeier,et al.  Subjective Workload Assessment in a Spatial Memory Task , 1984 .

[95]  Vernoi Battiste,et al.  Transport Pilot Workload: A Comparison of Two Subjective Techniques , 1988 .

[96]  I. Singh,et al.  Individual differences in monitoring failures of automation , 1993 .

[97]  Glenn F. Wilson,et al.  Physiological and Subjective Evaluation of a New Aircraft Display , 1990 .

[98]  R. W. Pew,et al.  Situational awareness in the commercial aircraft cockpit: a cognitive perspective , 1990, 9th IEEE/AIAA/NASA Conference on Digital Avionics Systems.

[99]  S G Hart,et al.  Inflight application of three pilot workload measurement techniques. , 1987, Aviation, space, and environmental medicine.

[100]  J. C. Byers,et al.  Comparison of Four Subjective Workload Rating Scales , 1992 .

[101]  R. T. Gill,et al.  The Effect of Acceleration Stress on Human Workload. , 1985 .

[102]  John F. Courtright,et al.  Use of SWAT in Usaf System T & E , 1984 .

[103]  W W Wierwille,et al.  Evaluation of 20 Workload Measures Using a Psychomotor Task in a Moving-Base Aircraft Simulator , 1983, Human factors.

[104]  Raja Parasuraman,et al.  Monitoring Automation Failures: Effects of Automation Reliability and Task Complexity , 1992 .

[105]  A. Kramer,et al.  Physiological metrics of mental workload: A review of recent progress , 1990, Multiple-task performance.

[106]  A. Rapoport,et al.  A Comparison of the Eigenvalue Method and The Geometric Mean Procedure for Ratio Scaling , 1986 .

[107]  George P. Boucek,et al.  Transport Aircraft Crew Workload Assessment-Where Have We Been and Where Are We Going? , 1987 .