Measuring Effectiveness in Digital Game-Based Learning: A Methodological Review

In recent years, a growing number of studies are being conducted into the effectiveness of digital game-based learning (DGBL). Despite this growing interest, there is a lack of sound empirical evidence on the effectiveness of DGBL due to different outcome measures for assessing effectiveness, varying methods of data collection and inconclusive or difficult to interpret results. This has resulted in a need for an overarching methodology for assessing the effectiveness of DGBL. The present study took a first step in this direction by mapping current methods used for assessing the effectiveness of DGBL. Results showed that currently, comparison of results across studies and thus looking at effectiveness of DGBL on a more general level is problematic due to diversity in and suboptimal study designs. Variety in study design relates to three issues, namely different activities that are implemented in the control groups, different measures for assessing the effectiveness of DGBL and the use of different statistical techniques for analyzing learning outcomes. Suboptimal study designs are the result of variables confounding study results. Possible confounds that were brought forward in this review are elements that are added to the game as part of the educational intervention (e.g., required reading, debriefing session), instructor influences and practice effects when using the same test pre- and post-intervention. Lastly, incomplete information on the study design impedes replication of studies and thus falsification of study results.

[1]  Theodore Schick.  Readings in the Philosophy of Science: From Positivism to Postmodernism , 1999 .

[2]  Bryan Piatkowski,et al.  Bringing Evolution to a Technological Generation: A Case Study with the Video Game SPORE , 2012 .

[3]  Alvin C. M. Kwan,et al.  Online vocabulary games as a tool for teaching and learning English vocabulary , 2006 .

[4]  John A. Ellis,et al.  An Evaluation of Three Computer-Based Instructional Strategies in Basic Electricity and Electronics Training , 2000 .

[5]  Fengfeng Ke,et al.  A Qualitative Meta-Analysis of Computer Games as Learning Tools , 2009 .

[6]  M. Taras,et al.  ASSESSMENT – SUMMATIVE AND FORMATIVE – SOME THEORETICAL REFLECTIONS , 2005 .

[7]  Lieven De Marez,et al.  Improving arithmetic skills through gameplay: Assessment of the effectiveness of an educational game in terms of cognitive and affective learning outcomes , 2014, Inf. Sci..

[8]  Geertje Bekebrede,et al.  A brief methodology for researching and evaluating serious games and game- based learning , 2014 .

[9]  J. D. Fletcher,et al.  Computer Games and Instruction. , 2011 .

[10]  Feng S. Din,et al.  The Effects of Playing Educational Video Games on Kindergarten Achievement , 2001 .

[11]  Robert A. Reiser,et al.  Trends and issues in instructional design and technology , 2002 .

[12]  Thorsten Gerber,et al.  Applied Longitudinal Data Analysis Modeling Change And Event Occurrence , 2016 .

[13]  Harri Ketamo,et al.  An Adaptive Geometry Game for Handheld Devices , 2003, J. Educ. Technol. Soc..

[14]  Andrew Moshirnia,et al.  The Educational Potential of Modified Video Games , 2007 .

[15]  Bert Quade,et al.  Advances in Physical Education , 2011 .

[16]  D. Dimitrov,et al.  Pretest-posttest designs and measurement of change. , 2003, Work.

[17]  E. Salas,et al.  Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. , 1993 .

[18]  Ruth Stevens,et al.  Improving children's working memory and classroom performance , 2010 .

[19]  Julián Moreno,et al.  Digital Competition Game to Improve Programming Skills , 2012, J. Educ. Technol. Soc..

[20]  Iván Martínez-Ortiz,et al.  Learning Analytics and Educational Games: Lessons Learned from Practical Experience , 2013, GALA.

[21]  Herre van Oostendorp,et al.  Current Practices in Serious Game Research: A Review from a Learning Outcomes Perspective , 2009 .

[22]  Wei Pan,et al.  Assessing the effectiveness of a 3-D instructional game on improving mathematics achievement and motivation of middle school students , 2012, Br. J. Educ. Technol..

[23]  V. Shute.  SteAlth ASSeSSment in computer-BASed GAmeS to Support leArninG , 2011 .

[24]  J. Bligh,et al.  Evaluating educational interventions , 1999, BMJ.

[25]  Ya-Ting Carolyn Yang,et al.  Building virtual cities, inspiring intelligent citizens: Digital games for developing students' problem solving and learning motivation , 2012, Comput. Educ..

[26]  Petra Kaufmann,et al.  Experimental And Quasi Experimental Designs For Research , 2016 .

[27]  Marina Papastergiou,et al.  Digital Game-Based Learning in high school Computer Science education: Impact on educational effectiveness and student motivation , 2009, Comput. Educ..

[28]  Adriana G. Bus,et al.  Effects of a brief but intensive remedial computer intervention in a sub-sample of kindergartners with early literacy delays , 2011, Reading and Writing.

[29]  E. Deci,et al.  Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. , 2000, Contemporary educational psychology.

[30]  Richard E. Mayer,et al.  Multimedia learning and games. , 2011 .

[31]  Henrik Schønau-Fog,et al.  “Sure, I Would Like to Continue” , 2012 .

[32]  R. Kozma.  Will media influence learning? Reframing the debate , 1994 .

[33]  M. Vauras,et al.  Instructing Low‐Achievers in Mathematical Word Problem Solving , 2010 .

[34]  Ángel Serrano-Lagunaa,et al.  Learning Analytics and Educational Games: Lessons Learned from Practical Experience , 2013 .

[35]  Fengfeng Ke.  Computer games application within alternative classroom goal structures: cognitive, metacognitive, and affective evaluation , 2008 .

[36]  Jan Van Looy,et al.  Poverty is not a game: behavioral changes and long term effects after playing PING , 2012 .

[37]  M. Gianluca,et al.  The Potential of Digital Games for Empowerment and Social Inclusion of Groups at Risk of Social and Economic Exclusion: Evidence and Opportunity for Policy , 2013 .

[38]  Linda Hutchinson,et al.  Evaluating and researching the effectiveness of educational interventions , 1999, BMJ.

[39]  Mark R. Leary,et al.  Introduction to behavioral research methods , 1990 .

[40]  C. Coles,et al.  Games that "work": using computer games to teach alcohol-affected children about fire and street safety. , 2007, Research in developmental disabilities.

[41]  Anne Anastasi,et al.  Differential Psychology: Individual and Group Differences in Behavior , 1958 .

[42]  Robert T Hays,et al.  The Effectiveness of Instructional Games: A Literature Review and Discussion , 2005 .

[43]  Deborah I Thompson,et al.  Playing for real: video games and stories for health-related behavior change. , 2008, American journal of preventive medicine.

[44]  S. W. Kim,et al.  Effectiveness of MMORPG-based instruction in elementary English education in Korea , 2010, J. Comput. Assist. Learn..

[45]  James E. Driskell,et al.  Games, Motivation, and Learning: A Research and Practice Model , 2002 .

[46]  E. Deci,et al.  Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. , 2000, The American psychologist.

[47]  J R Crawford,et al.  Demonstration of savings on the AVLT and development of a parallel form. , 1989, Journal of clinical and experimental neuropsychology.

[48]  R. Kanthan,et al.  The impact of specially designed digital games-based learning in undergraduate pathology and medical education. , 2011, Archives of pathology & laboratory medicine.

[49]  Rolf Kretschmann.  Digital Sport-Management Games and Their Contribution to Prospective Sport-Managers’ Competence Development , 2012 .

[50]  Cyril Brom,et al.  Turning High-Schools into Laboratories? Lessons Learnt from Studies of Instructional Effectiveness of Digital Games in the Curricular Schooling System , 2012, Edutainment.

[51]  Michael Barnett,et al.  Using Video Games to Support Pre-Service Elementary Teachers Learning of Basic Physics Principles , 2011 .

[52]  H. O'Neil,et al.  Classification of learning outcomes: evidence from the computer games literature , 2005 .

[53]  Thomas Connolly,et al.  Games-Based Learning Advancements for Multi-Sensory Human Computer Interfaces: Techniques and Effective Practices , 2009 .

[54]  Alexander Verbraeck,et al.  The Real-Time Global Supply Chain Game: New Educational Tool for Developing Supply Chain Management Professionals , 2006, Transportation Journal.

[55]  J. Singer,et al.  Applied Longitudinal Data Analysis , 2003 .

[56]  David J. Miller,et al.  Educational benefits of using game consoles in a primary classroom: A randomised controlled trial , 2011, Br. J. Educ. Technol..

[57]  Atsusi Hirumi,et al.  The effects of modern mathematics computer games on mathematics achievement and class motivation , 2010, Comput. Educ..

[58]  J. E. Korteling,et al.  Transfer of Gaming: Transfer of training in serious gaming , 2011 .

[59]  Maria Virvou,et al.  Combining Software Games with Education: Evaluation of its Educational Effectiveness , 2005, J. Educ. Technol. Soc..

[60]  Judith Calder,et al.  Programme Evaluation and Quality: A Comprehensive Guide to Setting Up an Evaluation System , 1994 .

[61]  Ximena López,et al.  Beyond Nintendo: design and assessment of educational video games for first and second grade students , 2003, Comput. Educ..

[62]  Karl R. Popper,et al.  Science : Conjectures and Refutations , 2005 .

[63]  David J. Miller,et al.  Using a games console in the primary classroom: Effects of 'Brain Training' programme on computation and self-esteem , 2010, Br. J. Educ. Technol..

[64]  Alain Lieury,et al.  Kawashima vs “Super Mario”! Should a game be serious in order to stimulate cognitive aptitudes? , 2010 .

[65]  Hassan Rastegarpour,et al.  The effect of card games and computer games on learning of chemistry concepts , 2012 .

[66]  Richard E. Ferdig,et al.  Handbook of Research on Effective Electronic Gaming in Education , 2009 .