Evaluating system utility and conceptual fit using CASSM

There is a wealth of user-centred evaluation methods (UEMs) to support the analyst in assessing interactive systems. Many of these support detailed aspects of use-for example: is the feedback helpful? Are labels appropriate? Is the task structure optimal? Few UEMs encourage the analyst to step back and consider how well a system supports users' conceptual understandings and system utility. In this paper, we present CASSM, a method, which focuses on the quality of 'fit' between users and an interactive system. We describe the methodology of conducting a CASSM analysis and illustrate the approach with three contrasting worked examples (a robotic arm, a digital library system and a drawing tool) that demonstrate different depths of analysis. We show how CASSM can help identify re-design possibilities to improve system utility. CASSM complements established evaluation methods by focusing on conceptual structures rather than procedures. Prototype tool support for completing a CASSM analysis is provided by Cassata, an open source development.

[1]  Mary Beth Rosson,et al.  Getting around the task-artifact cycle: how to make claims and design by scenario , 1992, TOIS.

[2]  Nigel Shadbolt,et al.  Use of the Critical Decision Method to Elicit Expert Knowledge: A Case Study in the Methodology of Cognitive Task Analysis , 1998, Hum. Factors.

[3]  Dennis R. Wixon.  Evaluating usability methods: why the current literature fails the practitioner , 2003, INTR.

[4]  Thomas R. G. Green,et al.  Cognitive dimensions of notations , 1990 .

[5]  Andrew M. Dearden,et al.  Enhancing Contextual Analysis to Support the Design of Development Tools , 2004, BCS HCI.

[6]  Andrew Howes,et al.  The Nature of Device Models: The Yoked State Space Hypothesis and Some Experiments With Text Editors , 1990, Hum. Comput. Interact..

[7]  Chauncey E. Wilson,et al.  The Usability Engineering Framework for Product Design and Evaluation , 1997 .

[8]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[9]  Robert C. Williges,et al.  Criteria For Evaluating Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[10]  Ann Blandford,et al.  CASSM and cognitive walkthrough: usability issues with ticket vending machines , 2004, Behav. Inf. Technol..

[11]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[12]  K. J. Vicente,et al.  Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work , 1999 .

[13]  Thomas R. G. Green.  The cognitive dimension of viscosity: A sticky problem for HCI , 1990, INTERACT.

[14]  Thomas P. Moran,et al.  Getting into a system: External-internal task mapping analysis , 1983, CHI '83.

[15]  Marian Petre,et al.  Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework , 1996, J. Vis. Lang. Comput..

[16]  Brian Fitzgerald,et al.  Formalized systems development methodologies: a critical perspective , 1996, Inf. Syst. J..

[17]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[18]  Philippe A. Palanque,et al.  Ontological Sketch Models: Highlighting User-System Misfits , 2003 .

[19]  Kasper Hornbæk,et al.  What do usability evaluators do in practice?: an explorative study of think-aloud testing , 2006, DIS '06.

[20]  A.,et al.  Cognitive Engineering , 2008, Encyclopedia of GIS.

[21]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[22]  Ann Blandford,et al.  Scoping Analytical Usability Evaluation Methods: A Case Study , 2008, Hum. Comput. Interact..

[23]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[24]  Bonnie E. John,et al.  Tracking the effectiveness of usability evaluation methods , 1997, Behav. Inf. Technol..

[25]  Ann Blandford,et al.  Formalising an Understanding of User-System Misfits , 2004, EHCI/DS-VIS.

[26]  Wayne D. Gray,et al.  Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods , 1998, Hum. Comput. Interact..

[27]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[28]  Ann Blandford,et al.  Concept-based Analysis of Surface and Structural Misfits (CASSM) Tutorial notes , 2004 .

[29]  Ann Blandford,et al.  Multiple viewpoints on computer supported team work: a case study on ambulance dispatch. , 2002 .

[30]  O. Bertelsen,et al.  Activity Theory , 2003 .

[31]  Richard M. Young,et al.  Programmable user models for predictive evaluation of interface designs , 1989, CHI '89.

[32]  John Millar Carroll.  HCI Models, Theories, and Frameworks: Toward a Multidisciplinary Science , 2003 .

[33]  T. Landauer,et al.  Handbook of Human-Computer Interaction , 1997 .

[34]  Cathleen Wharton,et al.  The cognitive walkthrough method: a practitioner's guide , 1994 .

[35]  Idris Hsi,et al.  Measuring the conceptual fitness of an application in a computing ecosystem , 2004, WISER '04.

[36]  Ann Blandford,et al.  Training software engineers in a novel usability evaluation technique , 1998, Int. J. Hum. Comput. Stud..

[37]  Ann Blandford,et al.  Specifying user knowledge for the design of interactive systems , 1996, Softw. Eng. J..

[38]  Keith Duncan,et al.  Cognitive Engineering , 2017, Encyclopedia of GIS.

[39]  Donald A. Norman,et al.  User Centered System Design , 1986 .

[40]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[41]  George R. S. Weir,et al.  People and Computers IX: Crafting Interaction: Styles, Metaphors, Modalities and Agents , 1994 .

[42]  David Benyon,et al.  The skull beneath the skin: entity-relationship models of information artifacts , 1996, Int. J. Hum. Comput. Stud..

[43]  Bonnie E. John,et al.  Learning and using the cognitive walkthrough method: a case study approach , 1995, CHI '95.