Abstract : The utility and effectiveness of simulation-based training of air combat skill has been supported by a long research history. However, despite this success, there has been much debate on the effectiveness of gradesheets that have been used as the primary measure of air combat skill. This study assessed the effectiveness of a Gradesheet that has been used to measure air combat team performance in Distributed Mission Operations (DMO). It was hypothesized that the DMO Gradesheet is an effective measure of air combat team performance, and is therefore sensitive to differences in team performance over time during training, among performance indicators, and across team experience levels. Between August 2000 and December 2001, 32 teams of F-16 pilots from various U.S. Air Force operational units participated in a week of structured DMO training. Training consisted of teams flying four networked high-fidelity F-16 simulators in scenarios against multiple constructed threats. On all scenarios of the training syllabus, air combat Subject Matter Experts (SMEs) graded the performance of the teams on 40 indicators that make constitute the DMO Gradesheet. Analyses of the data found that aggregate mean graded team performance increased over a week of DMO training. However, for all indicators, mean graded performance increased linearly and at approximately the same rate, suggesting that the Gradesheet captured only a single component of air combat performance. Furthermore, team experience did not moderate change in graded performance over missions as expected, undermining the claim that the Gradesheet measured even general performance. Results suggest that the DMO Gradesheet as used in the current study lacked the sensitivity, validity, and reliability desired in a measure of air combat performance. Future directions in the development of subjective and objective measures of air combat performance are discussed. (8 tables, 21 figures, 24 refs.)
[1]
Leslie A. Whitaker,et al.
An Information Processing Classification of Beyond-Visual-Range Air Intercepts
,
1993
.
[2]
Jan de Leeuw,et al.
Introducing Multilevel Modeling
,
1998
.
[3]
M. R. Houck,et al.
Tools for assessing situational awareness in an operational fighter environment.
,
1994,
Aviation, space, and environmental medicine.
[4]
Wayne L. Waag,et al.
Development of a Composite Measure for Predicting Engagement Outcome during Air Combat Maneuvering
,
1992
.
[5]
Byron J. Pierce,et al.
Performance Measurement Requirements for Tactical Aircrew Training
,
1987
.
[6]
Herbert H. Bell,et al.
Evaluating the Effectiveness of Flight Simulators for Training Combat Skills: A Review
,
1998
.
[7]
Keith A. Seaman.
Improving F-15C Air Combat Training with Distributed Mission Training (DMT) Advanced Simulation
,
1999
.
[8]
Michael J. Kelly.
Performance Measurement during Simulated Air-to-Air Combat
,
1988
.
[9]
Winston Bennett,et al.
Developing competency-based methods for near-real-time air combat problem solving assessment
,
2002,
Comput. Hum. Behav..
[10]
Anthony S. Bryk,et al.
Hierarchical Linear Models: Applications and Data Analysis Methods
,
1992
.
[11]
James McGuinness,et al.
Effectiveness Evaluation for Air Combat Training
,
1982
.
[12]
Winston Bennett,et al.
Using Distributed Mission Training to Augment Flight Lead Upgrade Training
,
2001
.