A systematic literature review of methodology used to measure effectiveness in digital game-based learning

In recent years, a growing number of studies is being conducted into the effectiveness of digital game-based learning (DGBL). Despite this growing interest, however, it remains difficult to draw general conclusions due to the disparities in methods and reporting. Guidelines or a standardized procedure for conducting DGBL effectiveness research would allow to compare results across studies and provide well-founded and more generalizable evidence for the impact of DGBL. This study presents a first step in this process by mapping current practices through a systematic literature review. The review included peer-reviewed journal and conference publications between 2000 and 2012. Other inclusion criteria were that (1) the study’s primary aim was effectiveness measurement of cognitive learning outcomes, (2) the focus was on digital games and (3) a pre-post design with a control group was used. Twenty-five publications were found eligible for this study. Important differences were found in the number of control groups used and the type of intervention implemented in the control group (e.g. traditional classroom teaching, use of multimedia, computer-based learning, paper exercises, other games, or no intervention). Regarding the implementation method of the DGBL intervention in the experimental group, two approaches can be distinguished: stand-alone intervention or as part of a larger program. Moreover, a wide variety of effectiveness measures was used: measures for learning outcomes were complemented with time measurements and/or with self-reported measurements for self-efficacy and motivation. Learning effect calculation also varied, introducing pre-test scores in the analysis, conducting a separate analysis on pre- and post-test scores or conducting an analysis on difference scores. Our study thus indicates that a variety of methods is being used in DGBL effectiveness research opening a discussion regarding the potential and requirements for future procedural guidelines.

[1]  Wei Pan,et al.  Assessing the effectiveness of a 3-D instructional game on improving mathematics achievement and motivation of middle school students , 2012, Br. J. Educ. Technol..

[2]  Ya-Ting Carolyn Yang,et al.  Building virtual cities, inspiring intelligent citizens: Digital games for developing students' problem solving and learning motivation , 2012, Comput. Educ..

[3]  Karl R. Popper,et al.  Science : Conjectures and Refutations , 2005 .

[4]  David J. Miller,et al.  Using a games console in the primary classroom: Effects of 'Brain Training' programme on computation and self-esteem , 2010, Br. J. Educ. Technol..

[5]  Alain Lieury,et al.  Kawashima vs “Super Mario”! Should a game be serious in order to stimulate cognitive aptitudes? , 2010 .

[6]  C. Coles,et al.  Games that "work": using computer games to teach alcohol-affected children about fire and street safety. , 2007, Research in developmental disabilities.

[7]  Anne Anastasi,et al.  Differential Psychology: Individual and Group Differences in Behavior , 1958 .

[8]  Van Looy Jan,et al.  State of play of digital games for empowerment and inclusion: a review of the literature and empirical cases , 2012 .

[9]  Feng S. Din,et al.  The Effects of Playing Educational Video Games on Kindergarten Achievement , 2001 .

[10]  Fengfeng Ke.  Computer games application within alternative classroom goal structures: cognitive, metacognitive, and affective evaluation , 2008 .

[11]  Alvin C. M. Kwan,et al.  Online vocabulary games as a tool for teaching and learning English vocabulary , 2006 .

[12]  Jan Van Looy,et al.  Poverty is not a game: behavioral changes and long term effects after playing PING , 2012 .

[13]  G. Salomon,et al.  4 No distribution without individuals ' cognition : a dynamic interactional view GAVRIEL SALOMON 5 Living knowledge : the social distribution of cultural resources for thinking , 2022 .

[14]  Mark R. Leary,et al.  Introduction to behavioral research methods , 1990 .

[15]  Amy Shirong Lu,et al.  A Scoping Review of Health Game Research: Past, Present, and Future. , 2012, Games for health journal.

[16]  James M. Boyle,et al.  A systematic literature review of empirical evidence on computer games and serious games , 2012, Comput. Educ..

[17]  Hassan Rastegarpour,et al.  The effect of card games and computer games on learning of chemistry concepts , 2012 .

[18]  Yavuz Akbulut,et al.  TRENDS AND ISSUES IN INSTRUCTIONAL DESIGN AND TECHNOLOGY , 2009 .

[19]  R. Kanthan,et al.  The impact of specially designed digital games-based learning in undergraduate pathology and medical education. , 2011, Archives of pathology & laboratory medicine.

[20]  Alexander Verbraeck,et al.  The Real-Time Global Supply Chain Game: New Educational Tool for Developing Supply Chain Management Professionals , 2006, Transportation Journal.

[21]  D. Dimitrov,et al.  Pretest-posttest designs and measurement of change. , 2003, Work.

[22]  Ruth Stevens,et al.  Improving children's working memory and classroom performance , 2010 .

[23]  Julián Moreno,et al.  Digital Competition Game to Improve Programming Skills , 2012, J. Educ. Technol. Soc..

[24]  David J. Miller,et al.  Educational benefits of using game consoles in a primary classroom: A randomised controlled trial , 2011, Br. J. Educ. Technol..

[25]  Atsusi Hirumi,et al.  The effects of modern mathematics computer games on mathematics achievement and class motivation , 2010, Comput. Educ..

[26]  Deborah I Thompson,et al.  Playing for real: video games and stories for health-related behavior change. , 2008, American journal of preventive medicine.

[27]  S. W. Kim,et al.  Effectiveness of MMORPG-based instruction in elementary English education in Korea , 2010, J. Comput. Assist. Learn..

[28]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[29]  Bryan Piatkowski,et al.  Bringing Evolution to a Technological Generation: A Case Study with the Video Game SPORE , 2012 .

[30]  H. Mandl,et al.  Inert knowledge: Analyses and remedies , 1996 .

[31]  P. Vorderer,et al.  Serious games : mechanisms and effects , 2009 .

[32]  Michael Barnett,et al.  Using Video Games to Support Pre-Service Elementary Teachers Learning of Basic Physics Principles , 2011 .

[33]  Henry Jenkins.  Confronting the Challenges of Participatory Culture: Media Education for the 21st Century , 2006 .

[34]  R. Gagne.  Learning outcomes and their effects: Useful categories of human performance. , 1984 .

[35]  Marina Papastergiou,et al.  Digital Game-Based Learning in high school Computer Science education: Impact on educational effectiveness and student motivation , 2009, Comput. Educ..

[36]  Adriana G. Bus,et al.  Effects of a brief but intensive remedial computer intervention in a sub-sample of kindergartners with early literacy delays , 2011, Reading and Writing.

[37]  J R Crawford,et al.  Demonstration of savings on the AVLT and development of a parallel form. , 1989, Journal of clinical and experimental neuropsychology.

[38]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[39]  James E. Driskell,et al.  Games, Motivation, and Learning: A Research and Practice Model , 2002 .

[40]  A. Whitehead.  The aims of education , 1929 .

[41]  Harri Ketamo,et al.  An Adaptive Geometry Game for Handheld Devices , 2003, J. Educ. Technol. Soc..

[42]  Andrew Moshirnia,et al.  The Educational Potential of Modified Video Games , 2007 .

[43]  R. Kozma.  Will media influence learning? Reframing the debate , 1994 .

[44]  M. Vauras,et al.  Instructing Low‐Achievers in Mathematical Word Problem Solving , 2010 .

[45]  Maria Virvou,et al.  Combining Software Games with Education: Evaluation of its Educational Effectiveness , 2005, J. Educ. Technol. Soc..

[46]  R. Clark.  Media will never influence learning , 1994 .

[47]  John A. Ellis,et al.  An Evaluation of Three Computer-Based Instructional Strategies in Basic Electricity and Electronics Training , 2000 .