A comparison of two analytical evaluation methods for educational computer games for young children

In this paper we describe a comparison of two analytical methods for educational computer games for young children. The methods compared in the study are the Structured Expert Evaluation Method (SEEM) and the Combined Heuristic Evaluation (HE) (based on a combination of Nielsen’s HE and the fun-related concepts from Malone and Lepper) with both usability and fun heuristics for children’s computer games. To verify SEEM’s relative quality, a study was set up in which adult evaluators predicted problems in computer games. Outcomes based on thoroughness (whether the analytical method finds all problems), validity (whether the analytical method uncovers problems that are likely to be true) and appropriateness (whether the method is applied correctly) are compared. The results show that both the thoroughness and validity of SEEM are higher than the thoroughness and validity of the Combined HE. The appropriateness scores indicate that SEEM gives evaluators more guidance when predicting problems than the Combined HE does.

[1]  Miguel Nussbaum,et al.  Playability in Action Videogames: A Qualitative Design Model , 2002, Hum. Comput. Interact..

[2]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[3]  Panos Markopoulos,et al.  Assessing the applicability of the structured expert evaluation method (SEEM) for a wider age group , 2006, IDC '06.

[4]  Thomas W. Malone,et al.  What makes things fun to learn? heuristics for designing instructional computer games , 1980, SIGSMALL '80.

[5]  Wolmet Barendregt,et al.  Empirical evaluation of usability and fun in computer games for children , 2003 .

[6]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[7]  Lynne Hall,et al.  Changing Analysts' Tunes: The Surprising Impact of a New Instrument for Usability Inspection Method Assessment , 2004 .

[8]  George R. S. Weir,et al.  People and Computers IX: Crafting Interaction: Styles, Metaphors, Modalities and Agents , 1994 .

[9]  Jacqueline Brodie,et al.  Applying user testing data to UEM performance metrics , 2004, CHI EA '04.

[10]  Gilbert Cockton,et al.  Inspection-based evaluations , 2002 .

[11]  Donna L. Cuomo,et al.  Understanding Usability Issues Addressed by Three User-System Interface Evaluation Techniques , 1994, Interact. Comput..

[12]  S. MacFarlane,et al.  Adapting the Heuristic Evaluation Method for Use with Children , 2006 .

[13]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[14]  Gilbert Cockton,et al.  Understanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation , 2001, BCS HCI/IHM.

[15]  Wolmet Barendregt,et al.  Predicting effectiveness of children participants in user testing based on personality characteristics , 2007, Behav. Inf. Technol..

[16]  Malcolm P. Atkinson,et al.  Comparison of evaluation methods using structured usability problem reports , 1997, Behav. Inf. Technol..

[17]  Robert C. Williges,et al.  Criteria For Evaluating Usability Evaluation Methods , 2003, Int. J. Hum. Comput. Interact..

[18]  Heather Desurvire,et al.  Using heuristics to evaluate the playability of games , 2004, CHI EA '04.

[19]  Wolmet Barendregt,et al.  Towards a Framework for Design Guidelines for Young Children?s Computer Games , 2004, ICEC.

[20]  Wolmet Barendregt,et al.  A Structured Expert Evaluation Method for the Evaluation of Children's Computer Games , 2005, INTERACT.

[21]  Andrew Sears,et al.  Heuristic Walkthroughs: Finding the Problems Without the Noise , 1997, Int. J. Hum. Comput. Interact..

[22]  W Barendregt,et al.  Developing a coding scheme for detecting usability and fun problems in computer games for young children , 2006, Behavior research methods.

[23]  Melissa A. Federoff,et al.  HEURISTICS AND USABILITY GUIDELINES FOR THE CREATION AND EVALUATION OF FUN IN VIDEO GAMES , 2002 .

[24]  Wolmet Barendregt,et al.  Evaluating Usability and Challenge during Initial and Extended Use of Children's Computer Games , 2004, BCS HCI.

[25]  Thomas W. Malone,et al.  What Makes Things Fun to Learn? A Study of Intrinsically Motivating Computer Games. , 1981 .

[26]  Randy J. Pagulayan,et al.  User-centered design in games , 2012 .

[27]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.