Evaluation Methods for the Effective Assessment of Simulation Games A Literature Review

Simulation games play an important role in the area of technology-based education. They allow the simulation of real-world problems in a risk-free environment and thereby intend to increase the learning experience of students. However, assessing the effectiveness of a simulation game is necessary to opti-mize elements of the game and increase the learning effect for students. For this, different evaluation methods exist, which do not always cover all phases when running a simulation game. In this study, we conduct a literature review to ana-lyze evaluation methods for the pre-game, in-game, and post-game assessment of simulation games. In accordance with our inclusion criteria, we selected 31 peer-reviewed articles, and categorized them according to a didactic framework that describes four phases of running simulation games: Preparation, Introduc-tion, Interaction and Conclusion phase. Based on the results, we provide a con-crete evaluation strategy that can be used to assess simulation games throughout all phases. This study contributes to theory by providing an overview of evalua-tion methods for the assessment of simulation games within the different game phases. It contributes to practice by providing a concrete evaluation strategy that can be adapted and used to assess simulation games.

[1]  Tobias Mettler,et al.  Serious Games as a Means for Scientific Knowledge Transfer—A Case From Engineering Management Education , 2015, IEEE Transactions on Engineering Management.

[2]  Jyoti Bhardwaj,et al.  Evaluation of the lasting impacts on employability of co-operative serious game-playing by first year Computing students: An exploratory analysis , 2014, 2014 IEEE Frontiers in Education Conference (FIE) Proceedings.

[3]  Abdelbaset Jamal Abdellatif,et al.  Serious games: Quality characteristics evaluation framework and case study , 2018, 2018 IEEE Integrated STEM Education Conference (ISEC).

[4]  Riccardo Berta,et al.  Assessment in and of Serious Games: An Overview , 2013, Adv. Hum. Comput. Interact..

[5]  Imed Boughzala,et al.  Learning Business Process Management through Serious Games: Feedbacks on the Usage of INNOV8 , 2016, 2016 IEEE 18th Conference on Business Informatics (CBI).

[6]  Hélène Michel,et al.  Characterizing Serious Games Implementation's Strategies: Is Higher Education the New Playground of Serious Games? , 2016, 2016 49th Hawaii International Conference on System Sciences (HICSS).

[7]  Mordechai Ben-Ari,et al.  Learning computer science concepts with Scratch , 2013, Comput. Sci. Educ..

[8]  Carlos Vaz de Carvalho,et al.  Is game-based learning suitable for engineering education? , 2012, Proceedings of the 2012 IEEE Global Engineering Education Conference (EDUCON).

[9]  Thomas M. Connolly,et al.  Evaluating Games-Based Learning , 2010, Int. J. Virtual Pers. Learn. Environ..

[10]  A. Lillard,et al.  Playful Learning and Montessori Education. , 2013 .

[11]  Helmut Krcmar,et al.  A Competency Model for "Industrie 4.0" Employees , 2017, Wirtschaftsinformatik.

[12]  Björn Niehaves,et al.  Reconstructing the giant: On the importance of rigour in documenting the literature search process , 2009, ECIS.

[13]  Dirk Ifenthaler,et al.  Assessment for Game-Based Learning , 2012 .

[14]  M. Al-Smadi,et al.  Assessment in serious games: An enhanced approach for integrated assessment forms and feedback to support guided learning , 2012, 2012 15th International Conference on Interactive Collaborative Learning (ICL).

[15]  Imed Boughzala,et al.  Characterization and Evaluation of Serious Games: A Perspective of Their Use in Higher Education , 2013, 2013 46th Hawaii International Conference on System Sciences.

[16]  Zacharoula Smyrnaiou,et al.  From Game to Guidance: The Innovative Evaluation Approach of the P4G Simulation Business Game , 2017, 2017 Fourth International Conference on Mathematics and Computers in Sciences and in Industry (MCSI).

[17]  Helmut Krcmar,et al.  Simulation games for the digital transformation of business processes: Development and application of two prototypes from the automotive and online retail sector , 2018, 2018 IEEE Global Engineering Education Conference (EDUCON).

[18]  Helmut Krcmar,et al.  Strengthening study skills by using ERPsim as a new tool within the Pupils' academy of serious gaming , 2016, 2016 IEEE Global Engineering Education Conference (EDUCON).

[19]  Greg Kearsley,et al.  Technology-Based Learning: Maximizing Human Performance and Corporate Success , 1998 .

[20]  Steven C. Gold,et al.  Developments in Business Gaming , 2009 .

[21]  Jannicke Baalsrud Hauge,et al.  State of the art of serious games for business and industry , 2011, 2011 17th International Conference on Concurrent Enterprising.

[22]  Matthew L. Jensen,et al.  Serious Games: An Evaluation Framework and Case Study , 2016, 2016 49th Hawaii International Conference on System Sciences (HICSS).

[23]  Christine Brown,et al.  Unpacking the potential of educational gaming: A new tool for gaming research , 2007 .

[24]  Dirk Ifenthaler,et al.  Alternative Assessment Strategies for Complex Problem Solving in Game-Based Learning Environments , 2011 .

[25]  Kristen E. DiCerbo Building the Evidentiary Argument in Game-Based Assessment. , 2017 .

[26]  Francesco Costantino,et al.  A simulation based game approach for teaching operations management topics , 2012, Proceedings Title: Proceedings of the 2012 Winter Simulation Conference (WSC).

[27]  Niall McShane,et al.  Mapping Learning and Game Mechanics for Serious Games Analysis in Engineering Education , 2017, IEEE Transactions on Emerging Topics in Computing.

[28]  Helmut Krcmar,et al.  The pupils' academy of serious gaming: Strengthening study skills with ERPsim , 2016, 2016 13th International Conference on Remote Engineering and Virtual Instrumentation (REV).

[29]  Shiffon Chatterjee,et al.  Computer Game-Based Learning and Pedagogical Contexts: Initial Findings from a Field Study , 2011, 2011 IEEE International Conference on Technology for Education.

[30]  Richard M. Hodgetts Management Gaming for Didactic Purposes , 1970 .

[31]  Matthias Christoph Utesch A Successful Approach to Study Skills: Go4C's Projects Strengthen Teamwork , 2016, Int. J. Eng. Pedagog..

[32]  Liz Boyle,et al.  The Implementation of Team Based Assessment In Serious Games , 2009, 2009 Conference in Games and Virtual Worlds for Serious Applications.

[33]  Ling-yuan Zeng An evaluation system of game-based learning based on data mining , 2012, Proceedings of 2012 2nd International Conference on Computer Science and Network Technology.

[34]  Lih-Hsing Hsu,et al.  On Developing the Learning Game for Graph Theory: A New Design Model Considering the Learners' Reflexiveness , 2016, 2016 5th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI).

[35]  Daniel L. Schwartz,et al.  Posterlet: A Game-Based Assessment of Children's Choices to Seek Feedback and to Revise , 2015, J. Learn. Anal..

[36]  Gregory A. Davis,et al.  Using a Retrospective Pre-Post Questionnaire To Determine Program Impact. , 2002 .

[37]  Catherine Cleophas,et al.  Designing serious games for revenue management training and strategy development , 2012, Proceedings Title: Proceedings of the 2012 Winter Simulation Conference (WSC).

[38]  Dion Hoe-Lian Goh,et al.  Participatory evaluation of an educational game for social skills acquisition , 2013, Comput. Educ..

[39]  Richard T. Watson,et al.  Analyzing the Past to Prepare for the Future: Writing a Literature Review , 2002, MIS Q..

[40]  Klaus-Dieter Thoben,et al.  On the effectiveness of teaching sustainable global manufacturing with serious gaming , 2013, 2013 International Conference on Engineering, Technology and Innovation (ICE) & IEEE International Technology Management Conference.

[41]  Iryna Zolotaryova,et al.  Serious games: Evaluation of the learning outcomes , 2016, 2016 13th International Conference on Modern Problems of Radio Engineering, Telecommunications and Computer Science (TCSET).

[42]  Aliane Loureiro Krassmann,et al.  Evaluation of Game-Based Learning Approaches through Digital Serious Games in Computer Science Higher Education: A Systematic Mapping , 2015, 2015 14th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames).

[43]  Donald C. Thatcher Promoting Learning through Games and Simulations , 1990 .

[44]  D. Kort,et al.  The Game Experience Questionnaire , 2013 .

[45]  Yuri Merkuryev,et al.  Business Simulation Game Development for Education and Training in Supply Chain Management , 2012, 2012 Sixth Asia Modelling Symposium.

[46]  Paula Escudeiro,et al.  Evaluation of Serious Games in Mobile Platforms with QEF: QEF (Quantitative Evaluation Framework) , 2012, 2012 IEEE Seventh International Conference on Wireless, Mobile and Ubiquitous Technology in Education.

[47]  Charlene Jennett,et al.  Learning When Serious: Psychophysiological Evaluation of a Technology-Enhanced Learning Game , 2014, J. Educ. Technol. Soc..