The Concordance of Subjective Measures and Military Pilot Performance in a study of Human-Interaction with Automation in a Complex Simulated Environment

Eight pilots with military experience completed a study of human-interaction with automation in a complex and dynamic air-to-ground search and destroy mission. Each pilot completed 32 ten-minute scenarios. Performance measures and subjective measures of mental workload, situation awareness, trust, confidence, and automation reliability were recorded after each scenario. An analysis of the agreement of the subjective measures between scenario types was performed. The results indicated that there was a high level of agreement between all of the subjective measures. Further, the subjective measures were similar, but not identical to the performance measures ranked by scenario type. Design implications are discussed.