Empirical Usability Testing in a Component-Based Environment: Improving Test Efficiency with Component-Specific Usability Measures

This paper addresses the issue of usability testing in a component-based software engineering environment, specifically measuring the usability of different versions of a component in a more powerful manner than other, more holistic, usability methods. Three component-specific usability measures are presented: an objective performance measure, a perceived ease-of-use measure, and a satisfaction measure. The objective performance measure is derived from the message exchange between components recorded in a log file, whereas the other measures are obtained through a questionnaire. The power of the measures was studied in an experimental setting. Eight different prototypes of a mobile telephone were subjected to usability tests, in which 80 subjects participated. Analyses of the statistical power of these measures show that the component-specific performance measure can be more powerful than overall usability measures, which means fewer users are needed in a test.

[1]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[2]  Joëlle Coutaz,et al.  PAC, an Object Oriented Model for Dialog Design , 1987 .

[3]  David E. Kieras,et al.  An Approach to the Formal Analysis of User Complexity , 1999, Int. J. Man Mach. Stud..

[4]  William L. Bewley,et al.  Human factors testing in the design of Xerox's 8010 “Star” office workstation , 1983, CHI '83.

[5]  Fabio Paternò.  Model-Based Design and Evaluation of Interactive Applications , 2000 .

[6]  Mark C. Detweiler,et al.  Alphabetic Input on a Telephone Keypad , 1990 .

[7]  M. M. Taylor,et al.  Layered Protocols for Computer-Human Dialogue, I: Principles , 1988, Int. J. Man Mach. Stud..

[8]  Jakob Nielsen.  Interact '87 , 1988, SGCH.

[9]  Stanley R. Parkinson,et al.  Computer display menus , 1983 .

[10]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[11]  Willem-Paul Brinkman,et al.  Usability Testing of Interaction Components: Taking the Message Exchange as a Measure of Usability , 2004, CADUI.

[12]  Bonnie E. John,et al.  Tracking the effectiveness of usability evaluation methods , 1997, Behav. Inf. Technol..

[13]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[14]  Fabio Paternò,et al.  Automatic Support for Usability Evaluation , 1998, IEEE Trans. Software Eng..

[15]  Anthony R. Hendrickson,et al.  Using Davis's Perceived Usefulness and Ease-of-use Instruments for Decision Making: A Confirmatory and Multigroup Invariance Analysis , 1998 .

[16]  Reinder Haakma.  Layered feedback in user-system interaction , 1998 .

[17]  Robert C. Williges,et al.  Collecting Detailed User Evaluations of Software Interfaces , 1985 .

[18]  Stephen Travis Pope,et al.  A cookbook for using the model-view controller user interface paradigm in Smalltalk-80 , 1988 .

[19]  Philip D. Gray,et al.  Instrumenting Bytecode for the Production of Usage Data , 2004, CADUI.

[20]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..