A formative evaluation of the training effectiveness of a computer game

Despite computer games and simulations' potential power in instruction and training, research on their training effectiveness for adults is limited, and a framework of evaluation is lacking. In addition, according to previous studies, a computer game may be one of the most effective tools to improve problem-solving. Problem-solving is one of the most significant competences whether in job settings or in schools, and as a result, teaching and assessing problem-solving become one of the most significant educational objectives (Mayer, 2002). However, there is limited research with adults that has shown the effects of game-based learning. In addition, according to previous researchers, a media's training effect such as a game is very limited when effective instructional strategies are lacking. In this dissertation, there was a pilot and a main study. In the pilot study, the researcher conducted a formative evaluation for a computer puzzle-solving game in terms of its effectiveness of enhancing both the learners' understanding about the content of the game and the strategies they adopted to solve the puzzles in the game. In the main study, thirty college and graduate students were asked to play the computer puzzle-solving game, i.e., SafeCracker, and were assessed before and after the two times of game-playing, using the knowledge mapping system and problem-solving strategy questions of retention and transfer. The results showed that the participants' problem-solving improved; i.e., their content understanding and problem-solving strategies were significantly increased after the game-playing. However, when compared to experts, the learning was low; i.e. students only achieved 4% of the content knowledge, 15% of the retention strategies, and 12% of the transfer strategies, representative of experts. The interjudge reliability of the problem-solving strategies measure was found satisfactory. In summary, although the learning was low, as expected, one outcome of this study is a research environment with reliable and valid measures of problem solving; that is, the knowledge maps as content understanding measure the retention and transfer questions as problem solving strategies measure, and the trait self-regulation questionnaire as self-regulation measure. Explanations of the findings are proposed, and implications for future research on games are discussed.