Evaluating user experience of adaptive digital educational games with Activity Theory

Adaptive digital educational games (DEGs) providing players with relevant interventions can enhance gameplay experience. This advance in game design, however, renders the user experience (UX) evaluation of DEGs even more challenging. To tackle this challenge, we developed a four-dimension evaluation framework (i.e., gaming experience, learning experience, adaptivity, and usability) and applied it to an empirical study with a DEG on teaching geography. Mixed-method approaches were adopted to collect data with 16 boys aged 10-11. Specifically, a so-called Dyadic User Experience Tests (DUxT) was employed; participants were paired up to assume different roles during gameplay. Learning efficacy was evaluated with a pre-post intervention measurement using a domain-specific questionnaire. Learning experience, gaming experiences and usability were evaluated with intensive in situ observations and interviews guided by a multidimensional scheme; content analysis of these transcribed audio data was supplemented by video analysis. Effectiveness of adaptivity algorithms was planned to be evaluated with automatic logfiles, which, unfortunately, could not be realised due to some technical problem. Nonetheless, the user-based data could offer some insights into this issue. Furthermore, we attempted to bridge the existing gap in UX research - the lack of theoretical frameworks in understanding user experience - by adopting Engestrom's (1987) extended framework of Activity Theory (AT) that provides contextual information essential for understanding contradictions and breakdowns observed in the interactions between the game players. The dyadic gameplay setting allows us to explore the issue of group UX. Implications for further applications of the AT framework in the UX research, especially the interplay between evaluation and redesign (i.e., downstream utility of UX evaluation methods), are discussed.

[1]  Bonnie E. John,et al.  Evaluating a multimedia authoring tool , 1997 .

[2]  K. Kuutti.  Activity theory as a potential framework for human-computer interaction research , 1995 .

[3]  Michele D. Dickey.  "Ninja Looting" for instructional design: the design challenges of creating a game-based learning environment , 2006, SIGGRAPH '06.

[4]  Aki Järvinen,et al.  Communication and Community in Digital Entertainment Services. Prestudy Research Report , 2002 .

[5]  T. Owen,et al.  Actor Network Theory and Social Science: possibilities and implications , 2011 .

[6]  Mikael B. Skov,et al.  Comparison of think-aloud and constructive interaction in usability testing with children , 2005, IDC '05.

[7]  Katherine Isbister,et al.  Game Usability - Advice from the Experts for Advancing the Player Experience , 2008 .

[8]  Virpi Roto,et al.  Understanding, scoping and defining user experience: a survey approach , 2009, CHI.

[9]  Jan Stage,et al.  The Interplay Between Usability Evaluation and User Interaction Design , 2006, Int. J. Hum. Comput. Interact..

[10]  Marko Turpeinen,et al.  The influence of implicit and explicit biofeedback in first-person shooter games , 2010, CHI.

[11]  Yvonne de Kort,et al.  Shared Fun Is Doubled Fun: Player Enjoyment as a Function of Social Setting , 2008, Fun and Games.

[12]  Nicoletta Di Blas,et al.  Educational Benefits: Testing and Evaluation of a Collaborative 3D World , 2005 .

[13]  A. N. Leont’ev,et al.  Activity, consciousness, and personality , 1978 .

[14]  J. Keller.  Motivational Design of Instruction , 1983 .

[15]  Franca Garzotto,et al.  Investigating the educational effectiveness of multiplayer online games for children , 2007, IDC.

[16]  Víctor M. González,et al.  The nature of managing multiple activities in the workplace , 2006 .

[17]  Yrjö Engeström,et al.  Enriching activity theory without shortcuts , 2008, Interact. Comput..

[18]  Andrew Burrow,et al.  Architectural designers and the interactive audience , 2005 .

[19]  B. Sutton-Smith,et al.  The Study of Games , 1971 .

[20]  J. Brine,et al.  Students' perceptions of a selected aspect of a computer mediated academic writing program: An activity theory analysis , 2006 .

[21]  Robert Rieger,et al.  Using mobile computing to enhance field study , 1997, CSCL.

[22]  Terry Winograd,et al.  Understanding computers and cognition - a new foundation for design , 1987 .

[23]  Antonella De Angeli,et al.  Towards a theory of user judgment of aesthetics and user interface quality , 2008, TCHI.

[24]  Rachel K. E. Bellamy,et al.  Designing educational technology: computer-mediated change , 1995 .

[25]  Karolien Poels,et al.  Measuring the experience of digital game enjoyment , 2008 .

[26]  Elke E. Mattheiss,et al.  Vicarious Learning with a Digital Educational Game: Eye-Tracking and Survey-Based Evaluation Approaches , 2010, USAB.

[27]  Y. Engeström,et al.  Developmental studies of work as a testbench of activity theory: The case of primary care medical practice , 1993 .

[28]  James Noble,et al.  Video game values: Human-computer interaction and games , 2007, Interact. Comput..

[29]  K. Poels,et al.  "It is always a lot of fun!": exploring dimensions of digital game experience using focus group methodology , 2007, Future Play.

[30]  Dietrich Albert,et al.  Knowledge Spaces: Theories, Empirical Research, and Applications , 1998 .

[31]  A. Pickering.  Science as practice and culture , 1992 .

[32]  Nick Lee,et al.  Otherness and the Actor Network , 1994 .

[33]  Victor Kaptelinin,et al.  From Human–Computer Interaction to Computer-Mediated Activity , 2000 .

[34]  Peta Wyeth,et al.  GameFlow: a model for evaluating player enjoyment in games , 2005, CIE.

[35]  Lorna Uden,et al.  Designing user interfaces using activity theory , 2001, Proceedings of the 34th Annual Hawaii International Conference on System Sciences.

[36]  Mary Jo Dondlinger,et al.  Educational Video Game Design: A Review of the Literature , 2007 .

[37]  Kurt Squire,et al.  Cultural Framing of Computer/Video Games , 2002, Game Stud..

[38]  Anders Drachen,et al.  Methods for Evaluating Gameplay Experience in a Serious Gaming Context , 2010, Int. J. Comput. Sci. Sport.

[39]  Effie Lai-Chong Law.  Evaluating the Downstream Utility of User Tests and Examining the Developer Effect: A Case Study , 2006, Int. J. Hum. Comput. Interact..

[40]  Paul van Schaik,et al.  Modelling user experience - An agenda for research and practice , 2010, Interact. Comput..

[41]  Lucy Suchman.  Plans and situated actions: the problem of human-machine communication , 1987 .

[42]  Randy J. Pagulayan,et al.  Designing for Fun: User-Testing Case Studies , 2005, Funology.

[43]  Sarah Diefenbach,et al.  Needs, affect, and interactive products - Facets of user experience , 2010, Interact. Comput..

[44]  Tim Marsh,et al.  Vicarious experience: staying there connected with and through our own and other characters , 2005 .

[45]  Maria Roussou,et al.  Examining young learners' activity within interactive virtual environments , 2004, IDC '04.

[46]  Natalia Juristo Juzgado,et al.  Interplay between usability and software development , 2010, J. Syst. Softw..

[47]  Martin Oliver,et al.  Activity theory and learning from digital games: developing an analytical methodology , 2006 .

[48]  Victor Kaptelinin,et al.  Computer-mediated activity: functional organs in social and developmental contexts , 1995 .

[49]  Victor Kaptelinin,et al.  Acting with technology: Activity theory and interaction design , 2006, First Monday.

[50]  S. Barab,et al.  Using Activity Theory to Conceptualize Online Community and Using Online Community to Conceptualize Activity Theory , 2004 .

[51]  James Paul Gee,et al.  What video games have to teach us about learning and literacy , 2007, CIE.

[52]  Maria Roussou,et al.  Exploring activity theory as a tool for evaluating interactivity and learning in virtual environments for children , 2008, Cognition, Technology & Work.

[53]  Eileen Scanlon,et al.  Using technology in Higher Education: an Activity Theory perspective , 2002, J. Comput. Assist. Learn..

[54]  Gitte Lindgaard,et al.  West meets East: Adapting Activity Theory for HCI & CSCW applications? , 2008, Interact. Comput..

[55]  Dietrich Albert,et al.  Micro Adaptive, Non-invasive Knowledge Assessment in Educational Games , 2008, 2008 Second IEEE International Conference on Digital Game and Intelligent Toy Enhanced Learning.

[56]  Kari Kuutti,et al.  Where are the Ionians of user experience research? , 2010, NordiCHI.

[57]  Marc Prensky,et al.  Digital game-based learning , 2000, CIE.

[58]  M. Csíkszentmihályi.  Flow: The Psychology of Optimal Experience , 1990 .

[59]  Xiaohui Liu,et al.  An Integrated Approach for Modeling Learning Patterns of Students in Web-Based Instruction: A Cognitive Style Perspective , 2008, TCHI.

[60]  Henk A. L. Kiers,et al.  Five (or three) robust questionnaire scale factors of personality without culture , 1991 .

[61]  Yvonne Rogers.  57 Varieties of Activity Theory , 2008, Interact. Comput..

[62]  B. Nardi.  Studying context: a comparison of activity theory, situated action models, and distributed cognition , 1995 .

[63]  Kasper Hornbæk,et al.  Exploring the Value of Usability Feedback Formats , 2009, Int. J. Hum. Comput. Interact..

[64]  Leah Hoffmann,et al.  Learning through games , 2009, Commun. ACM.

[65]  S. Freitas.  Learning in immersive worlds: A review of game-based learning , 2006 .

[66]  Charles M. Reigeluth,et al.  Instructional Design Theories and Models : An Overview of Their Current Status , 1983 .

[67]  Karen Schrier.  Using augmented reality games to teach 21st century skills , 2006, SIGGRAPH '06.

[68]  Regina Bernhaupt,et al.  Evaluating User Experience in Games - Concepts and Methods , 2010, Human-Computer Interaction Series.

[69]  Katja Battarbee,et al.  Co-experience: the social user experience , 2003, CHI Extended Abstracts.

[70]  L. Vygotsky.  Mind in Society: The Development of Higher Psychological Processes: Harvard University Press , 1978 .

[71]  Marc Hassenzahl,et al.  User experience - a research agenda , 2006, Behav. Inf. Technol..