Evaluation of multimedia applications using inspection methods: the Cognitive Walkthrough case

Abstract Many evaluation methods are to be found in research literature: they can be formal, automatic, empirical or informal. The informal methods include the so-called inspection methods, which provide a good compromise between the cost and implementation time on the one hand, and the results they make it possible to obtain on the other. Amongst these methods, Cognitive Walkthrough enables the detection of a certain number of usability defects and the estimation of the degree of seriousness of the defect. In this article, we concentrate on Cognitive Walkthrough. We are particularly interested in it because, as far as we know, it is the only method based on theory (the theory of learning through exploration, itself inspired by Norman's Action Theory). However, although its usefulness as regards software ergonomics has been recognised, its efficiency in the case of multimedia applications is still far from being proved and very few research projects have been published on the matter. In fact, multimedia documents have characteristics which differ from those of traditional human–machine systems. This article presents a study on the use of Cognitive Walkthrough for the evaluation of several multimedia applications intended for the general public; it reveals the difficulties met by users and the areas in which the method needs to be adapted.

[1]  Dominique L. Scapin,et al.  Ergonomic criteria for evaluating the ergonomic quality of interactive systems , 1997, Behav. Inf. Technol..

[2]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[3]  Keith Duncan,et al.  Cognitive Engineering , 2017, Encyclopedia of GIS.

[4]  Michael J. Kahn,et al.  Formal usability inspections , 1994 .

[5]  Brian Shackel,et al.  Human factors for informatics usability , 1991 .

[6]  T. Landauer,et al.  Handbook of Human-Computer Interaction , 1997 .

[7]  Julien Huart.  Mieux concevoir pour mieux communiquer à l'ère des nouveaux médias : vers des méthodes de conduite de projets et d'évaluation qualité de documents multimédias , 2000 .

[8]  B. Thomas,et al.  Usability Evaluation In Industry , 1996 .

[9]  Cathleen Wharton,et al.  The cognitive walkthrough method: a practitioner's guide , 1994 .

[10]  Robert A. Virzi.  Usability Inspection Methods , 1997 .

[11]  Franca Garzotto,et al.  A systematic method for hypermedia usability evaluation , 1997, New Rev. Hypermedia Multim..

[12]  Sylvie Leleu-Merviel,et al.  Vers une classification des procédés d'interactivité par niveaux corrélés aux données , 1997 .

[13]  Michael Cooper,et al.  Evaluating accessibility and usability of Web pages , 1999, ICCAD 1999.

[14]  Bonnie E. John,et al.  Learning and using the cognitive walkthrough method: a case study approach , 1995, CHI '95.

[15]  Seppo Pohjolainen,et al.  Pedagogical Principles for Evaluation of Hypermedia-Based Learning Environments in Mathematics , 1998, J. Univers. Comput. Sci..

[16]  Peter C. Wright,et al.  A Cost-Effective Evaluation Method for Use by Designers , 1991, Int. J. Man Mach. Stud..

[17]  Jean Vanderdonckt,et al.  Development milestones towards a tool for working with guidelines , 1999, Interact. Comput..

[18]  J. Schiller.  SECURE DISTRIBUTED COMPUTING , 1994 .

[19]  Randolph G. Bias,et al.  The pluralistic usability walkthrough: coordinated empathies , 1994 .

[20]  Jean Scholtz,et al.  Developing Usability Tools and Techniques for Designing and Testing Web Sites , 1998 .

[21]  Yufei Yuan,et al.  World Wide Web navigation aid , 2000, Int. J. Hum. Comput. Stud..

[22]  Jonas Löwgren,et al.  Knowledge-based evaluation as design support for graphical user interfaces , 1992, CHI.

[23]  Clare-Marie Karat,et al.  Comparison of empirical testing and walkthrough methods in user interface evaluation , 1992, CHI.

[24]  Andrew Sears,et al.  The effect of task description detail on evaluator performance with cognitive walkthroughs , 1998, CHI Conference Summary.

[25]  Martin C. Maguire,et al.  Evaluating User-Computer Interaction: A Framework , 1993, Int. J. Man Mach. Stud..

[26]  Jean Scholtz,et al.  Methods for Identifying Usability Problems with Web Sites , 1998, EHCI.

[27]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[28]  Morten Hertzum,et al.  The Evaluator Effect during First-Time Use of the Cognitive Walkthrough Technique , 1999, HCI.

[29]  Neil Bowers,et al.  Weblint: Quality Assurance for the World Wide Web , 1996, Comput. Networks.

[30]  Anne H. Anderson,et al.  Eye-Tracking Explorations in Multimedia Communications , 2001, BCS HCI/IHM.

[31]  Alain Durand.  Modélisation moléculaire : vers un nouvel outil d'aide à la conception multimédia , 1997 .

[32]  Ann Blandford,et al.  People and Computers XV - Interaction without Frontiers : Joint Proceedings of HCI 2001 and IHM 2001 , 2001 .

[33]  W. Wayt Gibbs,et al.  Software's Chronic Crisis , 1994 .

[34]  Rosemary J. Stevenson,et al.  Effects of Text Structure and Prior Knowledge of the Learner on Navigation in Hypertext , 1998, Hum. Factors.

[35]  Christophe Kolski,et al.  Vers la correction et la prévention des erreurs méthodologiques dans le cycle de vie d'applications multimédias , 1998 .

[36]  Marilyn Hughes Blackmon,et al.  Cognitive walkthrough for the web , 2002, CHI.

[37]  Christophe Kolski,et al.  A Rule-Based Approach to the Ergonomic "Static" Evaluation of Man-Machine Graphic Interface in Industrial Processes , 1991, Int. J. Man Mach. Stud..

[38]  Laurel Allender,et al.  Why choose? A process approach to usability testing , 1995 .

[39]  Clare-Marie Karat,et al.  A comparison of user interface evaluation methods , 1994 .

[40]  Pauline A. Smith.  Towards a Practical Measure of Hypertext Usability , 1996, Interact. Comput..

[41]  Kathy Buckner,et al.  A taxonomy of design guidance for hypermedia design , 1999, Interact. Comput..

[42]  Mark Chignell,et al.  Discount Video Analysis for Usability Engineering , 1995 .

[43]  Suma Suzuki,et al.  QUIS: applying a new walkthrough method to a product design process , 1999, HCI.

[44]  Christophe Kolski,et al.  Evaluation des interfaces homme-machine lors du développement des systèmes interactifs , 1996 .

[45]  Franca Garzotto,et al.  Hypermedia design, analysis, and evaluation issues , 1995, CACM.

[46]  Agnès Pollier.  Evaluation d'une interface par des ergonomes : diagnostics et strategies , 1991 .

[47]  Bernard Riera,et al.  A Questionnaire-based Discount Evaluation Method using Guidelines for Process Control Interactive Applications , 2000, TFWWG.

[48]  Nichole Simpson,et al.  Managing the use of style guides in an organisational setting: practical lessons in ensuring UI consistency , 1999, Interact. Comput..

[49]  Jurek Kirakowski,et al.  Measuring the Usability of Web Sites , 1998 .

[50]  George Casaday,et al.  Inspections and design reviews: framework, history and reflection , 1994 .

[51]  Daniel M. Wildman.  Getting the most from paired-user testing , 1995, INTR.

[52]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[53]  Jonathan Grudin,et al.  Utility and Usability: Research Issues and Development Contexts , 1992, Interact. Comput..

[54]  B. Senach.  Evaluation ergonomique des interfaces homme-machine : une revue de la littérature , 1990 .

[55]  Jane Webster,et al.  Perceived disorientation: an examination of a new measure to assess web design effectiveness , 2001, Interact. Comput..

[56]  Jean Vanderdonckt,et al.  Computer-Aided Design of User Interfaces II , 1999, Springer Netherlands.

[57]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[58]  Dominique L. Scapin,et al.  Ergonomic criteria for the evaluation of human-computer interfaces , 1993 .

[59]  Jochen Prümper,et al.  Software-Evaluation Based upon ISO 9241 Part 10 , 1993, VCHCI.

[60]  Sebastiano Bagnara,et al.  Multimedia systems in distance education: effects of usability on learning , 1999, Interact. Comput..

[61]  Philippe Trigano,et al.  A Tool for Evaluation Using Dynamic Navigation in a Set of Questions , 2000, TFWWG.

[62]  J. M. C. Bastien,et al.  L'ergonomie des sites web , 1998 .

[63]  David H. Jonassen,et al.  Designing Environments for Constructive Learning , 2012, NATO ASI Series.

[64]  Rachel Harrison,et al.  Applying Metrics to the Evaluation of Educational Hypermedia Applications , 1998, J. Univers. Comput. Sci..

[65]  Jeff Conklin,et al.  Hypertext: An Introduction and Survey , 1987, Computer.

[66]  Thomas S. Tullis,et al.  A system for evaluating screen formats: Research and application , 1988 .

[67]  Alain Lamboux-Durand.  La Modélisation moléculaire : Vers un nouvel outil d'aide à la conception multimédia , 1997 .

[68]  Jakob Nielsen,et al.  Guerrilla HCI: using discount usability engineering to penetrate the intimidation barrier , 1994 .

[69]  Dominique L. Scapin,et al.  The ergonomic criteria and the ISO/DIS 9241-10 dialogue principles: a pilot comparison in an evaluation task , 1999, Interact. Comput..

[70]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.