Evaluating an Adaptive Trainer for a Complex Decision-Making Task

Adaptive training (AT) is training that adjusts relative to a learner’s performance, aptitude, or learning preference (Landsberg, Van Buskirk, Astwood, Mercado, & Aakre, 2011). Several literature reviews have found that AT generally leads to positive learning outcomes (e.g., Durlach & Ray, 2011; Landsberg et al., 2011; McCarthy, 2008), but the AT literature lacks systematic empirical evidence to determine which AT techniques work best for which tasks and learners (Durlach & Ray, 2011; Vandewaetere, Desmet, & Clarebout, 2011). Therefore, the goal of the present experiment was to determine if similar AT techniques that were effective in one type of task would also be effective for a different task. Specifically, we used AT techniques (i.e., adapting feedback and scenario difficulty based on learner’s performance during training) that were shown to be effective for training periscope operators on a visuo-spatial task (Landsberg, Mercado, Van Buskirk, Lineberry, & Steinhauser, 2012) and applied them to a complex decision-making task to determine whether the effectiveness of these AT techniques are task-dependent. To test our research question, we developed the Adaptive Trainer for Joint Terminal Attack Controllers (ATTAC). ATTAC is a scenario-based training testbed that presents trainees with a series of close air support (CAS) situations and the trainee uses that information to develop a “game plan.” CAS is a highly complex military task, in which a Joint Terminal Attack Controller (JTAC) controls the maneuver of attacking aircraft used against hostile targets in close proximity of friendly forces. Due to the complex nature of this multi-step, multi-faceted mission, we chose to focus on one key decision-making step that sets the stage for the execution of a CAS mission – game plan development. When developing a game plan, JTACs must make decisions about how much control the JTAC will have over the attack, which weapons to employ, how the aircraft will observe the target, and the spacing between the attacking aircraft. The reported data are part of an ongoing data collection. To date, 33 U.S. Marine Corps personnel have participated in the experiment and were randomly assigned to one of three training conditions: AT, non-AT, and control. All trainees first completed a pre-test, in which they received nine scenarios similar to those used in ATTAC and had to determine a game plan. Next, during the training phase, trainees received training specific to their assigned condition. In the AT condition (n = 10), both feedback and scenario difficulty were adapted based on the trainee’s performance on the scenarios in ATTAC. In the non-AT condition (n = 11), the feedback and scenario difficulty stayed constant regardless of performance in ATTAC. In the control condition (n = 12), trainees received a short refresher on game plan development and did not complete any training scenarios in ATTAC. Finally, all participants completed the post-test that contained the same items as the pre-test but in a different order and this order was counterbalanced. The preliminary results comparing the pretest and post-test performance suggested that after only 35 minutes of training, there was a significant improvement in game plan development for participants in the AT (p < .001, Cohen’s d = 1.69) and non-AT conditions (p = .01, d = 0.77), while the control group did not improve (p = .14, d = 0.45). In addition, the AT condition showed significantly higher learning gains than the control condition (p = .04, d = 1.23), but there was no difference in learning gains between the AT and non-AT conditions (p = .27, d = 0.46) or the non-AT and control condition (p = .30, d = 0.42). The preliminary data show that AT leads to significantly higher learning gains compared to the control condition, which was meant to simulate students’ current training with game plan development (i.e., classroom lecture and graded simulation-based exercises). Currently, service members who are learning to conduct and participate in CAS missions do not have many low-stakes opportunities to practice game plan development outside of instructor-led exercises. Therefore, students who used the non-adaptive version of ATTAC may have benefited from exposure to a variety of CAS training scenarios, practice with the challenging decision-making process of developing a game plan, and feedback that displayed ideal game plans. However, it should be noted that learning gains did not differ between the non-AT condition and the control condition. The present experiment adds to the growing body of AT research by demonstrating the effectiveness of adapting scenario difficulty and feedback in a previously unexplored domain, which is a crucial step forward for the AT literature (Durlach & Ray, 2011). Additional research in this area is needed to determine when and how to invest in AT technologies to enhance and supplement classroom instruction. Identifying these best practices would lead to improvements in training for students by providing them with training that is suited to their individual needs. Likewise, these enhancements would benefit instructors by providing them additional classroom time with trainees that are “up to speed” to focus on more advanced topics.