Towards a Comprehensive Methodology for the Research and Evaluation of Serious Games

Abstract The author presents the methodological backgrounds and underlying research design of an on-going scientific research project concerned with the scientific evaluation of serious games and/or computer-based simulation-games (SG) for advanced learning. The main questions of this research project are: 1. what are the requirements and design principles for a comprehensive social-scientific methodology for the evaluation of SG? 2. To what extend does SG contribute to advanced learning? 3. What factors contribute to, or determine this learning? 4. To what extend and under what conditions can SG- based learning be transferred to the real world (RW)? Between 2004 and 2012, several hundreds of SG-sessions in the Netherlands with twelve different SG were evaluated systematically, uniformly and quantitatively to give a data-set of 2100 respondents in higher education and in work-organizations. The author presents the research model, the quasi-experimental design and evaluation instruments. This focus in this article is on methodology and data-set to establish a proper foundation for forthcoming publications on empirical results.

[1]  Igor S. Mayer,et al.  The Gaming of Policy and the Politics of Gaming: A Review , 2009 .

[2]  Christopher P. Barlett,et al.  Video Game Effects—Confirmed, Suspected, and Speculative , 2009 .

[3]  Kurt Squire,et al.  Cultural Framing of Computer/Video Games , 2002, Game Stud..

[4]  Geertje Bekebrede,et al.  Reviewing the need for gaming in education to accommodate the net generation , 2011, Comput. Educ..

[5]  Samantha A Adams,et al.  Use of "serious health games" in health care: a review. , 2010, Studies in health technology and informatics.

[6]  J. Cannon-Bowers,et al.  Establishing a Science of Game Based Learning , 2010 .

[7]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[8]  Frans Mäyrä,et al.  Power and control of games: children as the actors of game cultures , 2003, DiGRA Conference.

[9]  M. Csíkszentmihályi.  Flow: The Psychology of Optimal Experience , 1990 .

[10]  Leonard Reinecke,et al.  The Pleasures of Success: Game-Related Efficacy Experiences as a Mediator Between Player Performance and Game Enjoyment , 2011, Cyberpsychology Behav. Soc. Netw..

[11]  François Bousquet,et al.  An evolving simulation/gaming process to facilitate adaptive watershed management in northern mountainous Thailand , 2007 .

[12]  James E. Driskell,et al.  Games, Motivation, and Learning: A Research and Practice Model , 2002 .

[13]  G. Hofstede.  Cultural differences in teaching and learning , 1986 .

[14]  Nancy L. Burns-Sardone.  An investigation of the relationship between higher education learning environments and learner characteristics to the development of Information Technology fluency and course satisfaction , 2008 .

[15]  L. J. Carton,et al.  Gaming the future of an urban network , 2004 .

[16]  L. Hedman,et al.  Visuospatial skills and computer game experience influence the performance of virtual endoscopy , 2004, Journal of Gastrointestinal Surgery.

[17]  Mark D. Griffiths,et al.  Online Data Collection from Video Game Players: Methodological Issues , 2004, Cyberpsychology Behav. Soc. Netw..

[18]  Simon Egenfeldt-Nielsen.  Overview of research on the educational use of video games , 2006 .

[19]  David J. Brown,et al.  Can participating in games based learning improve mathematic skills in students with intellectual disabilities? , 2011, 2011 IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH).

[20]  James F. Knight,et al.  Serious gaming technology in major incident triage training: a pragmatic controlled trial. , 2010, Resuscitation.

[21]  N. Mackenzie,et al.  Research dilemmas: Paradigms, methods and methodology , 2006 .

[22]  Andrew J. Martin,et al.  Brief approaches to assessing task absorption and enhanced subjective experience: Examining ‘short’ and ‘core’ flow in diverse performance domains , 2008 .

[23]  Roland Pfister.  Gender Effects in Gaming Research: A Case for Regression Residuals? , 2011, Cyberpsychology Behav. Soc. Netw..

[24]  Fengfeng Ke,et al.  A Qualitative Meta-Analysis of Computer Games as Learning Tools , 2009 .

[25]  M. Kuit,et al.  Strategic Behavior and Regulatory Styles in the Netherlands Energy Industry , 2002 .

[26]  C. Greenblat.  Teaching with Simulation Games: A Review of Claims and Evidence. , 1973 .

[27]  Jean Écalle,et al.  Serious games as new educational tools: how effective are they? A meta-analysis of recent studies , 2013, J. Comput. Assist. Learn..

[28]  David W. Shaffer,et al.  Epistemic frames for epistemic games , 2006, Comput. Educ..

[29]  A. Edmondson.  Psychological Safety and Learning Behavior in Work Teams , 1999 .

[30]  F. Paas,et al.  A motivational perspective on the relation between mental effort and performance: Optimizing learner involvement in instruction , 2005 .

[31]  Brian F. Goldiez,et al.  Virtual Experience Test: A virtual environment evaluation questionnaire , 2010, 2010 IEEE Virtual Reality Conference (VR).

[32]  Geralien A. Holsbrink-Engels,et al.  Computer-Based Role-Playing for Interpersonal Skills Training , 1997 .

[33]  Herre van Oostendorp,et al.  Code Red: Triage, Or, COgnition-based DEsign Rules Enhancing Decisionmaking TRaining In A Game Environment , 2011 .

[34]  Daniel K. Mayes,et al.  Measuring Engagement in Video Games: A Questionnaire , 2001 .

[35]  Martin Shubik.  Political Economy, Oligopoly and Experimental Games , 1999 .

[36]  Alexandra Poulovassilis,et al.  Learning as immersive experiences: Using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world , 2010, Br. J. Educ. Technol..

[37]  Julia C. Lo,et al.  Integrated , Ecosystem-based Marine Spatial Planning: First Results from International Simulation-Ga , 2012 .

[38]  Thomas M. Connolly,et al.  Evaluating Games-Based Learning , 2010, Int. J. Virtual Pers. Learn. Environ..

[39]  K. Squire,et al.  Replaying history: learning world history through playing civilization iii , 2004 .

[40]  Christina M. Steiner,et al.  Non-invasive Assessment and Adaptive Interventions in Learning Games , 2009, 2009 International Conference on Intelligent Networking and Collaborative Systems.

[41]  Charles J. Kacmar,et al.  Development and Validation of the Political Skill Inventory , 2005 .

[42]  M. Ashton,et al.  Psychometric Properties of the HEXACO Personality Inventory , 2004, Multivariate behavioral research.

[43]  Wilfried Admiraal,et al.  The concept of flow in collaborative game-based learning , 2011, Comput. Hum. Behav..

[44]  E. V. D. Spek,et al.  Experiments in serious game design : a cognitive approach , 2011 .

[45]  J. Wolfe,et al.  Team Cohesion Effects on Business Game Performance , 1987 .

[46]  J. U. Hense,et al.  Theory-oriented evaluation for the design of and research in gaming and simulation , 2006 .

[47]  Elizabeth A. Boyle,et al.  Games for Learning: Does Gender Make a Difference? , 2009 .

[48]  Christine Armatas,et al.  Motivational Perspectives on Students’ Responses to Learning in Virtual Learning Environments , 2006 .

[49]  Tzyh-Lih Hsia,et al.  A study of student satisfaction in a blended e-learning system environment , 2010, Comput. Educ..

[50]  Fran C. Blumberg.  The effects of children's goals for learning on video game performance , 2000 .

[51]  Richard D. Blunt.  A causal-comparative exploration of the relationship between game-based learning and academic achievement: Teaching management with video games , 2006 .

[52]  Paul Jen-Hwa Hu,et al.  Technology-assisted learning: a longitudinal field study of knowledge category, learning effectiveness and satisfaction in language learning , 2007, J. Comput. Assist. Learn..

[53]  James M. Boyle,et al.  Engagement in digital entertainment games: A systematic review , 2012, Comput. Hum. Behav..

[54]  Casper Harteveld,et al.  Triadic Game Design - Balancing Reality, Meaning and Play , 2011 .

[55]  R. M. Brown,et al.  Gender and Video Game Performance , 1997 .

[56]  Gilad Ravid,et al.  Testing social theories in computer-mediated communication through gaming and simulation , 2006 .

[57]  Geertje Bekebrede,et al.  Experiencing Complexity: A gaming approach for understanding infrastructure systems , 2010 .

[58]  Martin Valcke,et al.  Do alternative instructional approaches result in different game performance learning outcomes? Authentic assessment in varying game conditions. , 2007 .

[59]  P. Vermeulen,et al.  Policy gaming for strategy and change , 2007 .

[60]  Benjamin M. Good,et al.  Games with a scientific purpose , 2011, Genome Biology.

[61]  Alessandro De Gloria,et al.  Enhancing the educational value of video games , 2009, CIE.

[62]  P. M. Kato,et al.  A Video Game Improves Behavioral Outcomes in Adolescents and Young Adults With Cancer: A Randomized Trial , 2008, Pediatrics.

[63]  R. Vallerand,et al.  On the Assessment of Situational Intrinsic and Extrinsic Motivation: The Situational Motivation Scale (SIMS) , 2000 .

[64]  D. Memmert,et al.  The Game Performance Assessment Instrument (GPAI): Some Concerns and Solutions for Further Development , 2008 .

[65]  M. Ashton,et al.  An Investigation of Personality Types within the HEXACO Personality Framework , 2009 .

[66]  Staffan Björk,et al.  Patterns In Game Design , 2004 .

[67]  D. M. Baba,et al.  Chapter 4 Determinants of Video Game Performance , 1993 .

[68]  R. Tafalla.  Gender Differences in Cardiovascular Reactivity and Game Performance Related to Sensory Modality in Violent Video Game Play1 , 2007 .

[69]  Douglas Williams,et al.  A research agenda for developing and implementing educational computer games , 2007, Br. J. Educ. Technol..

[70]  Andreas Maercker,et al.  E-health interventions for depression, anxiety disorders, dementia, and other disorders in old age: A review , 2011 .

[71]  Simon Egenfeldt-Nielsen,et al.  Beyond Edutainment: Exploring the Educational Potential of Computer Games , 2011 .

[72]  John Washbush,et al.  A Review of Scholarship on Assessing Experiential Learning Effectiveness , 2004 .

[73]  Richard M. Crowder,et al.  Validation of Serious Games Attributes Using the Technology Acceptance Model , 2010, 2010 Second International Conference on Games and Virtual Worlds for Serious Applications.

[74]  Clifford C Dacso,et al.  Assessing the Acceptability and Usability of an Interactive Serious Game in Aiding Treatment Decisions for Patients with Localized Prostate Cancer , 2011, Journal of medical Internet research.

[75]  Benjamin S. Bloom,et al.  Handbook On Formative and Summative Evaluation of Student Learning , 1971 .

[76]  H. H. Leemkuil,et al.  Is it all in the game? Learner support in an educational knowledge management simulation game , 2006 .

[77]  Alexander Toet,et al.  Engagement and EMG in Serious Gaming: Experimenting with Sound and Dynamics in the Levee Patroller Training Game , 2008, Fun and Games.

[78]  M. Wesson,et al.  Psychological collectivism: a measurement validation and linkage to group member performance. , 2006, The Journal of applied psychology.

[79]  Joel Brockner,et al.  Regulatory Focus Theory: Implications for the Study of Emotions at Work , 2001 .

[80]  Wen-Hao Huang,et al.  Evaluating learners' motivational and cognitive processing in an online game-based learning environment , 2011, Comput. Hum. Behav..

[81]  Alessandro De Gloria,et al.  Adaptive Experience Engine for Serious Games , 2009, IEEE Transactions on Computational Intelligence and AI in Games.

[82]  Linda L. Griffin,et al.  The game performance assessment instrument (GPAI): development and preliminary validation. , 1998 .

[83]  B. Steunenberg,et al.  Strategic Power in the European Union , 1999 .

[84]  David de Vaus,et al.  Research Design in Social Research , 2001 .

[85]  M. Salomon,et al.  Why Virtual-World Economies Matter , 2010 .

[86]  Marina Papastergiou,et al.  Digital Game-Based Learning in high school Computer Science education: Impact on educational effectiveness and student motivation , 2009, Comput. Educ..

[87]  F. Scharpf.  Games Real Actors Play: Actor-centered Institutionalism In Policy Research , 1997 .

[88]  Zahra Moussavi,et al.  Effects of an Interactive Computer Game Exercise Regimen on Balance Impairment in Frail Community-Dwelling Older Adults: A Randomized Controlled Trial , 2011, Physical Therapy.

[89]  D Duane Baldwin,et al.  Prior video game exposure does not enhance robotic surgical performance. , 2007, Journal of endourology.

[90]  Martin Oliver,et al.  How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? , 2006, Comput. Educ..

[91]  Thomas M. Connolly,et al.  The role of psychology in understanding the impact of computer games , 2011, Entertain. Comput..

[92]  J Dankelman,et al.  Serious gaming and voluntary laparoscopic skills training: A multicenter study , 2009, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[93]  June Lee,et al.  Effectiveness of Computer-Based Instructional Simulation: A Meta Analysis. , 1999 .

[94]  Jacquelyn L. Beyer,et al.  Simulation Review , 1978 .

[95]  James M. Boyle,et al.  A systematic literature review of empirical evidence on computer games and serious games , 2012, Comput. Educ..

[96]  Michael F. Young,et al.  Our Princess Is in Another Castle , 2012 .

[97]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.

[98]  Jeanne H. Brockmyer,et al.  The Development of the Game Engagement Questionnaire: A Measure of Engagement in Video Game Playing: Response to Reviews , 2009, Interacting with computers.

[99]  Kathleen-M. Lynn,et al.  Games for Civic Learning: A Conceptual Framework and Agenda for Research and Design , 2010, Games Cult..

[100]  Joel Bremson.  Using Gaming Simulation to Explore Long Range Fuel and Vehicle Transitions , 2012 .

[101]  Richard E. Mayer,et al.  The Cambridge Handbook of Multimedia Learning: Cognitive Theory of Multimedia Learning , 2005 .

[102]  Yoav Shoham,et al.  Essentials of game theory , 2008 .

[103]  Mable B. Kinzie,et al.  Gender differences in game activity preferences of middle school children: implications for educational game design , 2008 .

[104]  William Strein,et al.  Effects of Age and Visual-Motor Skills on Preschool Children's Computer-Game Performance. , 1987 .

[105]  C. Douglas Wetzel,et al.  The Effectiveness of Games for Educational Purposes: A Review of Recent Research , 1992 .

[106]  Jennifer Jenson,et al.  Gender, Simulation, and Gaming: Research Review and Redirections , 2010 .