Gamification for data gathering in emergency response exercises

Our paper describes how gamification can be implemented in an emergency response exercise. In particular, we focus on the potential of gamification to support self-evaluation processes through the automated gathering of data about the participants' performance. Disaster-exercises are typically constructed around a scenario posing more or less known challenges to the participants, with the performance often determined only by observation. While these observations certainly have a value in their own right, since participants are deeply engaged and immersed in such exercises (e.g. detailed descriptions of events, immediate interpretation of data, and placing results in context) they also have their limitations. Especially in regards to learning in emergency response exercises we encounter limitations due to the observer's cognitive limitations and the (limited) type of data that is being gathered. While external observations may be sufficient for evaluation the execution of tasks according to predetermined standards, less well defined and complex processes, such as decision making processes, are more difficult to evaluate. Such processes are for example influenced by cognitive biases, group dynamics and even political motives. Finally, on a more practical level, the dependence on external observers adds an additional requirement to already resource-intense disaster exercises, especially considering the required expertise. We aim to demonstrate the potential of gamification in disaster exercises to support the structured collection of data regarding the performance of the participants. This data would support self-evaluation, in turn reducing the dependence on external observers and provide additional insights. First we derive different constructs from the various learning objectives associated with complex tasks such as group decision making, coordination and building situational awareness. Next we translate these constructs in quantifiable measurements enabling data collection. Finally, we implement these measurements in gamification elements in a disaster exercise game. The resulting data is presented to the participants enabling them to reflect on their own performance as a team reducing the dependence on external observers. This concept has been explored, developed and studied in an experimental setting. Ten volunteers were involved in the study, participating in a simulated emergency response exercise at the University of Amsterdam. In this test the focus of the exercise and the implemented gamifications elements were the decision making process, information management and coordination efforts. For validation purposes, the participants were divided over three different groups each given the same tasks and challenges. Additional several experts in disaster exercises and decision making were present to compare the outcome of the groups' self-evaluations with their notes as 'traditional' observers. Our results indicate that gamification can be a strong tool to gather quantitative data concerning the learning goals of the exercise that in turn helps participants to evaluate their own performance for complex tasks during disaster responses. Participants were to a large extend able to indicate the same lessons as the observers. Demonstrating that the collected data supported the participants to reflect on their performance and identify improvements themselves. Moreover, participants also indicated additional lessons learned which were not noted by the observers.