Identifying usability and fun problems in a computer game during first use and after some practice

This paper describes an experiment to discover the change in the types of detected problems and the attitude of children towards a game when user testing a computer game for young children during first use and after they have practiced with a game. Both the numbers of different types of identified problems and the severity of the problems are investigated. Based on this knowledge, practitioners could adapt the set up of their user tests to effectively find as many aspects of the game as possible that merit change, according to the aims of the developers. The study shows that usability problems caused by a lack of knowledge were more often identified during first use. Furthermore, fun problems related to a too-high challenge level may disappear after some practice, whereas fun problems caused by the game taking over control for too long while the user wants to proceed playing the game were identified more often after some practice. The study shows that the impact severity of problems detected during first use was higher than when children had more practice with a game. As a result of these changes in experienced problems the commonly used measures efficiency, effectiveness and satisfaction increased when children had practiced with the game. Finally, the study also shows that the set of most severe problems identified during first use may be radically different from the set of most severe problems identified after some practice.

[1]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[2]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[3]  Donald A. Norman,et al.  User Centered System Design , 1986 .

[4]  Donald A. Norman,et al.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1988 .

[5]  Wolmet Barendregt,et al.  Evaluating Usability and Challenge during Initial and Extended Use of Children's Computer Games , 2004, BCS HCI.

[6]  Thomas W. Malone,et al.  What Makes Things Fun to Learn? A Study of Intrinsically Motivating Computer Games. , 1981 .

[7]  Carol M. Barnum,et al.  Usability testing and research , 2001 .

[8]  Wolmet Barendregt,et al.  Predicting effectiveness of children participants in user testing based on personality characteristics , 2007, Behav. Inf. Technol..

[9]  Gilbert Cockton,et al.  A Framework for Usability Problem Extraction , 1999, INTERACT.

[10]  Arnold P. O. S. Vermeeren,et al.  Managing the 'Evaluator Effect' in User Testing , 2003, INTERACT.

[11]  Melissa A. Federoff,et al.  HEURISTICS AND USABILITY GUIDELINES FOR THE CREATION AND EVALUATION OF FUN IN VIDEO GAMES , 2002 .

[12]  D. Norman.  Categorization of action slips. , 1981 .

[13]  Chuck Clanton,et al.  An interpreted demonstration of computer game design , 1998, CHI Conference Summary.

[14]  B. Bloom.  Taxonomy of educational objectives , 1956 .

[15]  Donald A. Norman,et al.  Some observations on mental models , 1987 .

[16]  Niels Jacobsen,et al.  Usability Evaluation Methods The Reliability and Usage of Cognitive Walkthrough and Usability Test , 1999 .

[17]  Edward L. Deci,et al.  The hidden costs of rewards , 1976 .

[18]  Jens Rasmussen,et al.  Human errors. a taxonomy for describing human malfunction in industrial installations , 1982 .

[19]  Jochen Prümper,et al.  Some surprising differences between novice and expert errors in computerized office work , 1992 .

[20]  Jochen Prümper,et al.  Errors in working with office computers: A first validation of a taxonomy for observed errors in a field setting , 1992, Int. J. Hum. Comput. Interact..

[21]  Randy J. Pagulayan,et al.  User-centered design in games , 2012 .

[22]  Morten Hertzum,et al.  The evaluator effect in usability tests , 1998, CHI Conference Summary.

[23]  Wolmet Barendregt,et al.  Empirical evaluation of usability and fun in computer games for children , 2003 .

[24]  M Kersten-Tsikalkina.  Evaluating usability and fun of children's products , 2001 .

[25]  J. Piaget.  Play, dreams and imitation in childhood , 1951 .

[26]  Huib de Ridder,et al.  DEVAN: A tool for detailed video analysis of user test data , 2002, Behav. Inf. Technol..

[27]  Mihaly Csikszentmihalyi,et al.  Intrinsic Rewards and Emergent Motivation , 1978 .

[28]  D. Berlyne.  Curiosity and exploration. , 1966, Science.

[29]  Colin Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[30]  Malcolm P. Atkinson,et al.  Comparison of evaluation methods using structured usability problem reports , 1997, Behav. Inf. Technol..

[31]  Jeffrey Rubin,et al.  Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests , 1994 .

[32]  Wolmet Barendregt,et al.  Towards a Framework for Design Guidelines for Young Children?s Computer Games , 2004, ICEC.

[33]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[34]  Richard Rouse,et al.  Game design : theory and practice , 2001 .

[35]  J. Read,et al.  Endurability, Engagement and Expectations: Measuring Children’s Fun , 2002 .

[36]  Jakob Nielsen,et al.  Estimating the number of subjects needed for a thinking aloud test , 1994, Int. J. Hum. Comput. Stud..