Savouring our mistakes: Learning from the FitQuest project

Abstract Although serious games for children can potentially have important social, educational and health benefits, the research process from initial game design to a robust evaluation is lengthy and complex. This paper describes the design and evaluation process of an exergame for children. It reports on the inconclusive results of a cluster randomised controlled trial (RCT) conducted among children aged 10–11 years attending 10 state-funded primary schools in Scotland. One class in each school was randomly allocated to intervention ( n = 5 , 111 children) or control ( n = 5 , 104 children). Intervention schools were given FitQuest, a smartphone game for the Android platform, and were requested to play the game during at least one hour of mandated Physical Education (PE) lessons per week for 5 weeks. Participants in the control arm took part in standard mandated PE lessons. Primary outcome measures were step count, minutes spent in moderate to vigorous physical activity (MVPA) and exercise self-efficacy. None of the children spent the recommended time per week playing FitQuest. There were no significant differences in step count, MVPA or self-efficacy by intervention group. The paper reflects on possible flaws during the design and evaluation process which could have led to the disappointing results, and presents some proposals for improving the research process for developing serious games for children. These include: deepening the ways in which we interact with domain expert colleagues, developing a shared understanding of the expectations for different phases of evaluation, closing the gap between game design knowledge and domain theories, raising the standards of evidence for design guidelines, encouraging synthesis across studies by evaluating mid-range theories rather than individual games, and developing guidelines for monitoring intervention fidelity in this domain.

[1]  Jack T. Harvey,et al.  Using a Socioecological Approach to Examine Participation in Sport and Physical Activity Among Rural Adolescent Girls , 2009, Qualitative health research.

[2]  Wanda Pratt,et al.  How to evaluate technologies for health behavior change in HCI research , 2011, CHI.

[3]  Shlomo Berkovsky,et al.  Physical Activity Motivating Games , 2009, AICS.

[4]  May O. Lwin,et al.  The efficacy of exergames-incorporated physical education lessons in influencing drivers of physical activity: A comparison of children and pre-adolescents , 2012 .

[5]  William W. Gaver What should we expect from research through design? , 2012, CHI.

[6]  John Zimmerman,et al.  Research through design as a method for interaction design research in HCI , 2007, CHI.

[7]  Sean A. Munson,et al.  Exploring goal-setting, rewards, self-monitoring, and sharing to motivate physical activity , 2012, 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops.

[8]  James A. Landay,et al.  Design requirements for technologies that encourage physical activity , 2006, CHI.

[9]  Olga De Troyer,et al.  Supporting the requirement analysis phase for the development of serious games for children , 2014, Int. J. Child Comput. Interact..

[10]  Joe Marshall,et al.  Expanding exertion gaming , 2016, Int. J. Hum. Comput. Stud..

[11]  Shlomo Berkovsky,et al.  Recommender algorithms in activity motivating games , 2010, RecSys '10.

[12]  Joshua H. West,et al.  Apps of Steel: Are Exercise Apps Providing Consumers With Realistic Expectations? , 2013, Health education & behavior : the official publication of the Society for Public Health Education.

[13]  Pelle Ehn,et al.  Scandinavian Design: On Participation and Skill , 1992, Usability - Turning Technologies into Tools.

[14]  Regan L. Mandryk,et al.  Gemini: A Pervasive Accumulated Context Exergame , 2011, ICEC.

[15]  Carol L. O'Donnell Defining, Conceptualizing, and Measuring Fidelity of Implementation and Its Relationship to Outcomes in K–12 Curriculum Intervention Research , 2008 .

[16]  John L. Sherry,et al.  Orientations to Video Games Among Gender and Age Groups , 2010 .

[17]  I. Janssen,et al.  Systematic review of the health benefits of physical activity and fitness in school-aged children and youth , 2010, The international journal of behavioral nutrition and physical activity.

[18]  M. Montola Exploring the Edge of the Magic Circle : Defining Pervasive Games , 2005 .

[19]  E. Biddiss,et al.  Active video games to promote physical activity in children and youth: a systematic review. , 2010, Archives of pediatrics & adolescent medicine.

[20]  Judith Good,et al.  Issues and Methods for Involving Young People in Design , 2013 .

[21]  Panos Markopoulos,et al.  Head Up Games: combining the best of both worlds by merging traditional and digital play , 2010, Personal and Ubiquitous Computing.

[22]  Víctor M. González,et al.  Mobile games and design requirements to increase teenagers' physical activity , 2012, Pervasive Mob. Comput..

[23]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[24]  Luc Geurts,et al.  Editorial: Learning from failures in game design for children , 2014, Int. J. Child Comput. Interact..

[25]  Andrew D. Miller,et al.  Designing pervasive health games for sustainability, adaptability and sociability , 2012, FDG.

[26]  A. Daley Can Exergaming Contribute to Improving Physical Activity Levels and Health Outcomes in Children? , 2009, Pediatrics.

[27]  Berry Eggen,et al.  Designing playful interactions for social interaction and physical play , 2010, Personal and Ubiquitous Computing.

[28]  S. Leatherdale,et al.  Active Video Games and Health Indicators in Children and Youth: A Systematic Review , 2013, PloS one.

[29]  Barbara Resnick,et al.  A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. , 2005, Journal of consulting and clinical psychology.

[30]  Richard M. Ryan,et al.  Glued to Games: How Video Games Draw Us In and Hold Us Spellbound , 2011 .

[31]  P. Bliese Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. , 2000 .

[32]  Luca Chittaro,et al.  Adaptation of Graphics and Gameplay in Fitness Games by Exploiting Motion and Physiological Sensors , 2007, Smart Graphics.

[33]  Zan Gao,et al.  A meta‐analysis of active video games on health outcomes among children and adolescents , 2015, Obesity reviews : an official journal of the International Association for the Study of Obesity.

[34]  Thomas J. Cook,et al.  Avoiding type III error in program evaluation: Results from a field experiment , 1980 .

[35]  D. Dunstan,et al.  A cluster-randomized controlled trial to reduce sedentary behavior and promote physical activity and health of 8-9 year olds: The Transform-Us! Study , 2011, BMC public health.

[36]  Joe Marshall,et al.  Misrepresentation of Health Research in Exertion Games Literature , 2017, CHI.

[37]  Silvia Lindtner,et al.  Fish'n'Steps: Encouraging Physical Activity with an Interactive Computer Game , 2006, UbiComp.

[38]  S. Blair Physical inactivity: the biggest public health problem of the 21st century. , 2009, British journal of sports medicine.

[39]  M. Petticrew,et al.  Developing and evaluating complex interventions: the new Medical Research Council guidance , 2008, BMJ : British Medical Journal.

[40]  T. C. Nicholas Graham,et al.  Design of an exergaming station for children with cerebral palsy , 2012, CHI.

[41]  Carol Dezateux,et al.  How active are our children? Findings from the Millennium Cohort Study , 2013, BMJ Open.

[42]  Andrew Peter Macvean,et al.  Understanding the exergame user experience: users' motivation, attitude and behaviour in a location-aware pervasive exergame for adolescent children , 2013 .

[43]  James Fogarty,et al.  Game design principles in everyday fitness applications , 2008, CSCW.

[44]  M. Six Silberman,et al.  When the implication is not to design (technology) , 2011, CHI.

[45]  Asimina Vasalou,et al.  Bridging serious games and participatory design , 2014, Int. J. Child Comput. Interact..

[46]  T. C. Nicholas Graham,et al.  Using games to increase exercise motivation , 2007, Future Play.

[47]  R. Plotnikoff,et al.  Physical Activity and Physical Self-Concept in Youth: Systematic Review and Meta-Analysis , 2014, Sports Medicine.

[48]  Eric Rosenbaum,et al.  Examining values: an analysis of nine years of IDC research , 2011, IDC.

[49]  Roope Raisamo,et al.  Seriously fun: exploring how to combine promoting health awareness and engaging gameplay , 2008, MindTrek '08.

[50]  Mae Keary,et al.  The Science of Evaluation: A Realist Manifesto , 2014, Online Inf. Rev..

[51]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[52]  Kim Delbaere,et al.  A Randomized Controlled Pilot Study of Home-Based Step Training in Older People Using Videogame Technology , 2013, PloS one.

[53]  Jih-Hsuan Lin,et al.  Using Active Video Games for Physical Activity Promotion , 2013, Health education & behavior : the official publication of the Society for Public Health Education.

[54]  M. Dobbins,et al.  Cochrane review: School‐based physical activity programs for promoting physical activity and fitness in children and adolescents aged 6‐18 , 2009 .

[56]  Alessandro De Gloria,et al.  Mapping learning and game mechanics for serious games analysis , 2015, Br. J. Educ. Technol..

[57]  J. Robertson,et al.  Understanding the Importance of Context: A Qualitative Study of a Location-Based Exergame to Enhance School Childrens Physical Activity , 2016, PloS one.

[58]  C. Abraham,et al.  A taxonomy of behavior change techniques used in interventions. , 2008, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[59]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[60]  Predrag V. Klasnja,et al.  Mind the theoretical gap: interpreting, using, and developing behavioral theory in HCI research , 2013, CHI.

[61]  Adam Fletcher,et al.  Realist randomised controlled trials: a new approach to evaluating complex public health interventions. , 2012, Social science & medicine.