Using the cognitive walkthrough to improve the design of a visual programming experiment

Visual programming languages aim to promote usability, but are rarely examined for it. One reason is the difficulty of designing successful experimental evaluations. We propose the cognitive walkthrough as an aid to improve experimental designs. This is a novel application of an HCI-derived technique designed for evaluating interfaces rather than experiments. The technique focuses on the potential difficulties of novice users and is therefore particularly suited for evaluating the programming situation, which is knowledge-based and non-routine. We describe an empirical study performed without benefit of a walkthrough and show how the study was improved by a series of walkthroughs. We found the method to be quick to use, effective at improving the experimental design, and usable by non-specialists.

[1]  Marian G. Williams,et al.  Comparison of visual and textual languages via task modeling , 1999, Int. J. Hum. Comput. Stud..

[2]  Thomas R. G. Green,et al.  Cognitive dimensions of notations , 1990 .

[3]  Allen Newell,et al.  The keystroke-level model for user performance time with interactive systems , 1980, CACM.

[4]  Margaret M. Burnett,et al.  Graphical definitions: expanding spreadsheet languages through direct manipulation and gestures , 1998, TCHI.

[5]  YangSherry,et al.  Forms/3: A first-order visual language to explore the boundaries of the spreadsheet paradigm , 2001 .

[6]  Margaret M. Burnett,et al.  Forms/3: A first-order visual language to explore the boundaries of the spreadsheet paradigm , 2001, Journal of Functional Programming.

[7]  Nick Wilde,et al.  Using the programming walkthrough to aid in programming language design , 1994, Softw. Pract. Exp..

[8]  Marian Petre,et al.  Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework , 1996, J. Vis. Lang. Comput..

[9]  Alan F. Blackwell,et al.  Metacognitive theories of visual programming: what do we think we are doing? , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[10]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[11]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[12]  Robin Jeffries,et al.  Applying cognitive walkthroughs to more complex user interfaces: experiences, issues, and recommendations , 1992, CHI.

[13]  Cathleen Wharton,et al.  Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces , 1990, CHI '90.

[14]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[15]  Bonnie E. John,et al.  Learning and using the cognitive walkthrough method: a case study approach , 1995, CHI '95.

[16]  Marita Franzke,et al.  Usability evaluation with the cognitive walkthrough , 1995, CHI 95 Conference Companion.

[17]  Wayne D. Gray,et al.  Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods , 1998, Hum. Comput. Interact..

[18]  Wayne Citrin,et al.  Testing effectiveness of algorithm animation , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[19]  Andrew Sears,et al.  The effect of task description detail on evaluator performance with cognitive walkthroughs , 1998, CHI Conference Summary.

[20]  P MoranThomas,et al.  The keystroke-level model for user performance time with interactive systems , 1980 .

[21]  Gregg Rothermel,et al.  WYSIWYT testing in the spreadsheet paradigm: an empirical evaluation , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[22]  Gregg Rothermel,et al.  What you see is what you test: a methodology for testing form-based visual programs , 1998, Proceedings of the 20th International Conference on Software Engineering.