The research and evaluation of serious games: Toward a comprehensive methodology

The authors present the methodological background to and underlying research design of an ongoing research project on the scientific evaluation of serious games and/or computer-based simulation games (SGs) for advanced learning. The main research questions are: (1) what are the requirements and design principles for a comprehensive social scientific methodology for the evaluation of SGs?; (2) to what extent do SGs contribute to advanced learning?; (3) what factors contribute to or determine this learning?; and (4) to what extent and under what conditions can SG-based learning be transferred to the real world? In the Netherlands between 2005 and 2012, several hundred SG sessions with 12 SGs were evaluated systematically, uniformly and quantitatively to create a dataset, which comprises data on 2488 respondents in higher education or work organizations. The authors present the research model, the quasi-experimental design and the evaluation instruments. This focus in this paper is on the methodology and dataset, which form a sound foundation for forthcoming publications on the empirical results. [ABSTRACT FROM AUTHOR]

[1]  Joel Brockner,et al.  Regulatory Focus Theory: Implications for the Study of Emotions at Work , 2001 .

[2]  Catholijn M. Jonker,et al.  Agent-Based Simulation of the Trust and Tracing Game for Supply Chains and Networks , 2008, J. Artif. Soc. Soc. Simul..

[3]  K. Squire,et al.  Replaying history: learning world history through playing civilization iii , 2004 .

[4]  Charles J. Kacmar,et al.  Development and Validation of the Political Skill Inventory , 2005 .

[5]  N. Léchopier.  " Experimental and quasi-experimental designs for research on teaching ", de Donald T. Campbell & Julian C. Stanley, (1963). , 2011 .

[6]  J. Cannon-Bowers,et al.  Establishing a Science of Game Based Learning , 2010 .

[7]  Gert Jan Hofstede,et al.  Why Simulation Games Work-In Search of the Active Substance: A Synthesis , 2010 .

[8]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[9]  Jean Écalle,et al.  Serious games as new educational tools: how effective are they? A meta-analysis of recent studies , 2013, J. Comput. Assist. Learn..

[10]  Yehia Mortagy,et al.  A Longitudinal Comparative Study of Student Perceptions in Online Education , 2010 .

[11]  Linda L. Griffin,et al.  The game performance assessment instrument (GPAI): development and preliminary validation. , 1998 .

[12]  Kathleen-M. Lynn,et al.  Games for Civic Learning: A Conceptual Framework and Agenda for Research and Design , 2010, Games Cult..

[13]  K. Perreault,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2011 .

[14]  Geertje Bekebrede,et al.  The Climate Game: Connecting Water Management and Spatial Planning through Simulation Gaming? , 2016 .

[15]  Wen-Hao Huang,et al.  Evaluating learners' motivational and cognitive processing in an online game-based learning environment , 2011, Comput. Hum. Behav..

[16]  M. Ashton,et al.  Psychometric Properties of the HEXACO Personality Inventory , 2004, Multivariate behavioral research.

[17]  Clint A. Bowers,et al.  Serious Game Design and Development: Technologies for Training and Learning , 2010 .

[18]  Wilfried Admiraal,et al.  The concept of flow in collaborative game-based learning , 2011, Comput. Hum. Behav..

[19]  E. V. D. Spek,et al.  Experiments in serious game design : a cognitive approach , 2011 .

[20]  J. Wolfe,et al.  Team Cohesion Effects on Business Game Performance , 1987 .

[21]  Richard Mayer,et al.  Multimedia Learning , 2001, Visible Learning Guide to Student Achievement.

[22]  Julia C. Lo,et al.  Integrated, ecosystem-based Marine Spatial Planning: Design and results of a game-based, quasi-experiment , 2013 .

[23]  Geertje Bekebrede,et al.  Learning in a game-based virtual environment: a comparative evaluation in higher education , 2013 .

[24]  Leonard Reinecke,et al.  The Pleasures of Success: Game-Related Efficacy Experiences as a Mediator Between Player Performance and Game Enjoyment , 2011, Cyberpsychology Behav. Soc. Netw..

[25]  Christine Armatas,et al.  Motivational Perspectives on Students’ Responses to Learning in Virtual Learning Environments , 2006 .

[26]  J. Dutton,et al.  Learning behaviours in the workplace: The role of high-quality interpersonal relationships and psychological safety , 2009 .

[27]  Valerie J. Shute,et al.  Conceptual Framework for Modeling , Assessing and Supporting Competencies within Game Environments , 2010 .

[28]  L. J. Carton,et al.  Gaming the future of an urban network , 2004 .

[29]  L. Hedman,et al.  Visuospatial skills and computer game experience influence the performance of virtual endoscopy , 2004, Journal of Gastrointestinal Surgery.

[30]  Fengfeng Ke,et al.  A Qualitative Meta-Analysis of Computer Games as Learning Tools , 2009 .

[31]  David J. Brown,et al.  Can participating in games based learning improve mathematic skills in students with intellectual disabilities? , 2011, 2011 IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH).

[32]  David W. Shaffer,et al.  Epistemic frames for epistemic games , 2006, Comput. Educ..

[33]  Pirkko Vartiainen,et al.  Evaluation Methods and Comparative Study , 2000 .

[34]  Joel Bremson.  Using Gaming Simulation to Explore Long Range Fuel and Vehicle Transitions , 2012 .

[35]  Richard E. Mayer,et al.  The Cambridge Handbook of Multimedia Learning: Cognitive Theory of Multimedia Learning , 2005 .

[36]  Geralien A. Holsbrink-Engels,et al.  Computer-Based Role-Playing for Interpersonal Skills Training , 1997 .

[37]  Fran C. Blumberg.  The effects of children's goals for learning on video game performance , 2000 .

[38]  François Bousquet,et al.  An evolving simulation/gaming process to facilitate adaptive watershed management in northern mountainous Thailand , 2007 .

[39]  Herre van Oostendorp,et al.  Code Red: Triage, Or, COgnition-based DEsign Rules Enhancing Decisionmaking TRaining In A Game Environment , 2011 .

[40]  Mark D. Griffiths,et al.  Online Data Collection from Video Game Players: Methodological Issues , 2004, Cyberpsychology Behav. Soc. Netw..

[41]  David de Vaus,et al.  Research Design in Social Research , 2001 .

[42]  James F. Knight,et al.  Serious gaming technology in major incident triage training: a pragmatic controlled trial. , 2010, Resuscitation.

[43]  Roland Pfister.  Gender Effects in Gaming Research: A Case for Regression Residuals? , 2011, Cyberpsychology Behav. Soc. Netw..

[44]  F. Paas,et al.  A motivational perspective on the relation between mental effort and performance: Optimizing learner involvement in instruction , 2005 .

[45]  Brian F. Goldiez,et al.  Virtual Experience Test: A virtual environment evaluation questionnaire , 2010, 2010 IEEE Virtual Reality Conference (VR).

[46]  Rudy McDaniel,et al.  The role teachers' expectations and value assessments of video games play in their adopting and integrating them into their classrooms , 2011, Br. J. Educ. Technol..

[47]  J. McGonigal.  Reality Is Broken: Why Games Make Us Better and How They Can Change the World , 2011 .

[48]  M. Kuit,et al.  Strategic Behavior and Regulatory Styles in the Netherlands Energy Industry , 2002 .

[49]  R. Vallerand,et al.  On the Assessment of Situational Intrinsic and Extrinsic Motivation: The Situational Motivation Scale (SIMS) , 2000 .

[50]  D. Memmert,et al.  The Game Performance Assessment Instrument (GPAI): Some Concerns and Solutions for Further Development , 2008 .

[51]  Richard E. Mayer,et al.  Cognitive Theory of Multimedia Learning , 2021, The Cambridge Handbook of Multimedia Learning.

[52]  R. Hyman.  Quasi-Experimentation: Design and Analysis Issues for Field Settings (Book) , 1982 .

[53]  Earl R. Babbie,et al.  The practice of social research , 1969 .

[54]  Ken Binmore,et al.  Fun and games : a text on game theory , 1991 .

[55]  James M. Boyle,et al.  A systematic literature review of empirical evidence on computer games and serious games , 2012, Comput. Educ..

[56]  Geertje Bekebrede,et al.  Experiencing Complexity: A gaming approach for understanding infrastructure systems , 2010 .

[57]  James D. Wright,et al.  Handbook of Survey Research. , 1985 .

[58]  Martin Valcke,et al.  Do alternative instructional approaches result in different game performance learning outcomes? Authentic assessment in varying game conditions. , 2007 .

[59]  H. H. Leemkuil,et al.  Is it all in the game? Learner support in an educational knowledge management simulation game , 2006 .

[60]  Alexander Toet,et al.  Engagement and EMG in Serious Gaming: Experimenting with Sound and Dynamics in the Levee Patroller Training Game , 2008, Fun and Games.

[61]  M. Wesson,et al.  Psychological collectivism: a measurement validation and linkage to group member performance. , 2006, The Journal of applied psychology.

[62]  Donald L. Kirkpatrick,et al.  Seven keys to unlock the four levels of evaluation , 2006 .

[63]  V. Shute,et al.  Melding the Power of Serious Games and Embedded Assessment to Monitor and Foster Learning: Flow and Grow , 2009 .

[64]  A. Edmondson.  Psychological Safety and Learning Behavior in Work Teams , 1999 .

[65]  Michael F. Young,et al.  Our Princess Is in Another Castle , 2012 .

[66]  Alessandro De Gloria,et al.  Building a Comprehensive R&D Community on Serious Games , 2012, VS-GAMES.

[67]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.

[68]  Jeanne H. Brockmyer,et al.  The Development of the Game Engagement Questionnaire: A Measure of Engagement in Video Game Playing: Response to Reviews , 2009, Interacting with computers.

[69]  John W. Creswell,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2010 .

[70]  P. Vermeulen,et al.  Policy gaming for strategy and change , 2007 .

[71]  Benjamin M. Good,et al.  Games with a scientific purpose , 2011, Genome Biology.

[72]  Douglas Williams,et al.  A research agenda for developing and implementing educational computer games , 2007, Br. J. Educ. Technol..

[73]  Simon Egenfeldt-Nielsen,et al.  Beyond Edutainment: Exploring the Educational Potential of Computer Games , 2011 .

[74]  Daniel K. Mayes,et al.  Measuring Engagement in Video Games: A Questionnaire , 2001 .

[75]  Martin Shubik.  Political Economy, Oligopoly and Experimental Games , 1999 .

[76]  D Duane Baldwin,et al.  Prior video game exposure does not enhance robotic surgical performance. , 2007, Journal of endourology.

[77]  Martin Oliver,et al.  How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? , 2006, Comput. Educ..

[78]  James M. Boyle,et al.  Engagement in digital entertainment games: A systematic review , 2012, Comput. Hum. Behav..

[79]  M. Salomon,et al.  Why Virtual-World Economies Matter , 2010 .

[80]  Marina Papastergiou,et al.  Digital Game-Based Learning in high school Computer Science education: Impact on educational effectiveness and student motivation , 2009, Comput. Educ..

[81]  F. Scharpf.  Games Real Actors Play: Actor-centered Institutionalism In Policy Research , 1997 .

[82]  Zahra Moussavi,et al.  Effects of an Interactive Computer Game Exercise Regimen on Balance Impairment in Frail Community-Dwelling Older Adults: A Randomized Controlled Trial , 2011, Physical Therapy.

[83]  M. Ashton,et al.  An Investigation of Personality Types within the HEXACO Personality Framework , 2009 .

[84]  Staffan Björk,et al.  Patterns In Game Design , 2004 .

[85]  D. M. Baba,et al.  Chapter 4 Determinants of Video Game Performance , 1993 .

[86]  Geertje Bekebrede,et al.  Veerkracht - a game for servant-leadership development , 2012 .

[87]  Benjamin S. Bloom,et al.  Handbook On Formative and Summative Evaluation of Student Learning , 1971 .

[88]  Willy Christian Kriz,et al.  Bridging the Gap , 2009 .

[89]  Clint A. Bowers,et al.  Serious Games Usability Testing: How to Ensure Proper Usability, Playability, and Effectiveness , 2011, HCI.

[90]  Alexandra Poulovassilis,et al.  Learning as immersive experiences: Using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world , 2010, Br. J. Educ. Technol..

[91]  Mark Stansfield,et al.  Towards the Development of a Games-Based Learning Evaluation Framework , 2009 .

[92]  Thomas M. Connolly,et al.  Evaluating Games-Based Learning , 2010, Int. J. Virtual Pers. Learn. Environ..

[93]  Mable B. Kinzie,et al.  Gender differences in game activity preferences of middle school children: implications for educational game design , 2008 .

[94]  C. Douglas Wetzel,et al.  The Effectiveness of Games for Educational Purposes: A Review of Recent Research , 1992 .

[95]  Thomas M. Connolly,et al.  The role of psychology in understanding the impact of computer games , 2011, Entertain. Comput..

[96]  J Dankelman,et al.  Serious gaming and voluntary laparoscopic skills training: A multicenter study , 2009, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[97]  Casper Harteveld,et al.  Triadic Game Design - Balancing Reality, Meaning and Play , 2011 .

[98]  Jennifer Jenson,et al.  Gender, Simulation, and Gaming: Research Review and Redirections , 2010 .

[99]  Dieter Schmidtchenb,et al.  Strategic power in the European Union: Evaluating the distribution of power in policy games , 1998 .

[100]  June Lee,et al.  Effectiveness of Computer-Based Instructional Simulation: A Meta Analysis. , 1999 .

[101]  Glenda A. Gunter,et al.  Factors Affecting Adoption of Video Games in the Classroom. , 2011 .

[102]  John Washbush,et al.  A Review of Scholarship on Assessing Experiential Learning Effectiveness , 2004 .

[103]  Richard M. Crowder,et al.  Validation of Serious Games Attributes Using the Technology Acceptance Model , 2010, 2010 Second International Conference on Games and Virtual Worlds for Serious Applications.

[104]  Clifford C Dacso,et al.  Assessing the Acceptability and Usability of an Interactive Serious Game in Aiding Treatment Decisions for Patients with Localized Prostate Cancer , 2011, Journal of medical Internet research.

[105]  Christina M. Steiner,et al.  Non-invasive Assessment and Adaptive Interventions in Learning Games , 2009, 2009 International Conference on Intelligent Networking and Collaborative Systems.

[106]  J. U. Hense,et al.  Theory-oriented evaluation for the design of and research in gaming and simulation , 2006 .

[107]  Elizabeth A. Boyle,et al.  Games for Learning: Does Gender Make a Difference? , 2009 .

[108]  P. M. Kato,et al.  A Video Game Improves Behavioral Outcomes in Adolescents and Young Adults With Cancer: A Randomized Trial , 2008, Pediatrics.

[109]  Andreas Maercker,et al.  E-health interventions for depression, anxiety disorders, dementia, and other disorders in old age: A review , 2011 .

[110]  Bente Tobiesen Meyer.  Comparative studies in game-based language learning: a discussion of methodology , 2010 .

[111]  Nicole Fruehauf.  Flow The Psychology Of Optimal Experience , 2016 .

[112]  M. Wittrock.  Handbook of research on teaching , 1986 .

[113]  Hilary Hollingsworth.  Cultural differences in teaching and learning , 2013 .

[114]  Christopher P. Barlett,et al.  Video Game Effects—Confirmed, Suspected, and Speculative , 2009 .

[115]  Samantha A Adams,et al.  Use of "serious health games" in health care: a review. , 2010, Studies in health technology and informatics.

[116]  C. Greenblat.  Teaching with Simulation Games: A Review of Claims and Evidence. , 1973 .

[117]  Hans B. Thorelli.  Thirty-Something Years of International Business Simulations , 1994 .

[118]  Igor S. Mayer,et al.  The Gaming of Policy and the Politics of Gaming: A Review , 2009 .

[119]  Kurt Squire,et al.  Cultural Framing of Computer/Video Games , 2002, Game Stud..

[120]  Geertje Bekebrede,et al.  Reviewing the need for gaming in education to accommodate the net generation , 2011, Comput. Educ..

[121]  Robert T Hays,et al.  The Effectiveness of Instructional Games: A Literature Review and Discussion , 2005 .

[122]  Clark C. Abt,et al.  Serious games , 2016, Springer International Publishing.

[123]  Frans Mäyrä,et al.  Power and control of games: children as the actors of game cultures , 2003, DiGRA Conference.

[124]  James E. Driskell,et al.  Games, Motivation, and Learning: A Research and Practice Model , 2002 .

[125]  Andrew J. Martin,et al.  Brief approaches to assessing task absorption and enhanced subjective experience: Examining ‘short’ and ‘core’ flow in diverse performance domains , 2008 .

[126]  Richard D. Duke,et al.  Gaming: The Future's Language , 2014 .

[127]  Donovan A. McFarlane,et al.  Evaluating Training Programs: The Four Levels , 2006 .

[128]  Yoav Shoham,et al.  Essentials of game theory , 2008 .

[129]  V. Shute.  SteAlth ASSeSSment in computer-BASed GAmeS to Support leArninG , 2011 .

[130]  Richard D. Blunt,et al.  A causal-comparative exploration of the relationship between game-based learning and academic achievement: Teaching management with video games , 2006 .

[131]  R. M. Brown,et al.  Gender and Video Game Performance , 1997 .

[132]  Casper Harteveld,et al.  Making Sense of Virtual Risks: A Quasi-Experimental Investigation into Game-Based Training , 2013 .

[133]  Magy Seif El-Nasr,et al.  The Effect of Age, Gender, and Previous Gaming Experience on Game Play Performance , 2010, HCIS.

[134]  Gilad Ravid,et al.  Testing social theories in computer-mediated communication through gaming and simulation , 2006 .

[135]  Olav Eikeland,et al.  Condescending ethics and action research , 2006 .

[136]  Simon Egenfeldt-Nielsen.  Overview of research on the educational use of video games , 2006 .

[137]  Fatimah Kari,et al.  Ocean and Coastal Management , 2014 .

[138]  N. Mackenzie,et al.  Research dilemmas: Paradigms, methods and methodology , 2006 .