An Examination of Four User-Based Software Evaluation Methods

Abstract Usability has become an important factor in the design of interactive software. In the quest to develop usable software, user participation in the design and evaluation process has been advocated. Four methods; logged data, questionnaires, interviews, and verbal protocol analyses have been the prominent base evaluation methods used, sometimes being combined in an attempt to provide a comprehensive evaluation. However, little is known about the strengths and weaknesses of these methods. In this study, these four methods were used to evaluate three different software types (spreadsheet, word processor, and database) with 148 subjects participating. The data obtained enabled a number of conclusions to be drawn regarding the usefulness of the methods when used individually and in combination.

[1]  Motoei Azuma,et al.  Experiments on Human-Computer Interaction Evaluation , 1987, Interacción.

[2]  Jacqueline V Laws,et al.  Video analysis in cognitive ergonomics: a methodological perspective. , 1989, Ergonomics.

[3]  W. David Penniman,et al.  Monitoring and evaluation of on-line information system usage , 1980, Inf. Process. Manag..

[4]  Jagdish C. Agrawal,et al.  Empirical Foundations of Information and Software Science IV , 1987, Springer US.

[5]  Mary Corbett,et al.  SUMI: the Software Usability Measurement Inventory , 1993, Br. J. Educ. Technol..

[6]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[7]  Henning Andersen,et al.  Tasks, errors, and mental models , 1988 .

[8]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[9]  C W Clegg,et al.  Towards a practical method of user interface evaluation. , 1989, Applied ergonomics.

[10]  Dan Diaper,et al.  Task Analysis for Human-Computer Interaction , 1990 .

[11]  John Karat,et al.  Software Evaluation Methodologies , 1988 .

[12]  Alan S. Neal,et al.  Playback: A Method for Evaluating the Usability of Software and Its Documentation , 1984, IBM Syst. J..

[13]  E. N. Corlett,et al.  Evaluation of human work : a practical ergonomics methodology , 1991 .

[14]  Seymour Sudman,et al.  Improving Interview Method and Questionnaire Design. , 1982 .

[15]  Stephen R. Ruth,et al.  The use of menus in the design of on-line sytems: a retrospective view , 1985, SGCH.

[16]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[17]  Peter Buckle.  BookNew methods in applied ergonomics: J. Wilson, E.N. Corlett and I. Manenica (Eds) Taylor and Francis Ltd, Basingstoke, 1987, ISBN 0-85066-384-9, pp 283 + x, £25.00 , 1988 .

[18]  Ron Henderson,et al.  A comparison of the four prominent user-based methods for evaluating the usability of computer software , 1995 .

[19]  Donna L. Cuomo.  A method for assessing the usability of graphical, direct-manipulation style interfaces , 1994, Int. J. Hum. Comput. Interact..

[20]  Mary Corbett,et al.  Effective Methodology for the Study of HCI , 1990 .

[21]  Steve Draper,et al.  Questionnaires as a software evaluation tool , 1983, CHI '83.

[22]  B. Schneiderman,et al.  Designing the User Interface. Strategies for Effective Human-Computer Interaction , 1992 .

[23]  Alistair Sutcliffe,et al.  Human-Computer Interface Design , 1988, Macmillan Computer Science Series.

[24]  David Meister.  Human Factors Testing and Evaluation , 1986 .

[25]  Martin Maguire,et al.  System monitoring: garbage generator or basis for comprehensive evaluation system? , 1990 .

[26]  Ken Eason,et al.  Towards the experimental study of usability , 1984 .

[27]  Elizabeth S. Cordingley,et al.  Knowledge elicitation techniques for knowledge-based systems , 1989 .

[28]  K. D. Duncan,et al.  Verbal reports: a problem in research design , 1988 .

[29]  F. H. Lochovsky,et al.  User Interface Design , 1985 .

[30]  METHODOLOGIES EMPLOYED IN THE PSYCHOLOGICAL EVALUATION OF H.C.I. , 1987 .

[31]  J. Robson,et al.  The role of user's perceived control in interface design, employing verbal protocol analysis. , 1989, Applied ergonomics.

[32]  E HoiemD,et al.  統合されたデータ収集・分析ツールの設計と利用 取り組みと考察 , 1994 .

[33]  Lisanne Bainbridge,et al.  Verbal protocol analysis. , 1990 .

[34]  Jochen Prümper,et al.  Some surprising differences between novice and expert errors in computerized office work , 1992 .

[35]  Gavriel Salvendy.  Cognitive engineering in the design of human-computer interaction and expert systems , 1987 .

[36]  T. Landauer,et al.  Handbook of Human-Computer Interaction , 1997 .

[37]  Marcia Guttentag,et al.  Handbook of Evaluation Research , 1975 .

[38]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[39]  D. Diaper,et al.  Knowledge Elicitation: Principles, Techniques and Applications , 1989 .

[40]  Derek Hoiem,et al.  Designing and using integrated data collection and analysis tools: challenges and considerations , 1994, Behav. Inf. Technol..

[41]  Janice Anne Rohn.  The usability engineering laboratories at Sun Microsystems , 1994, Behav. Inf. Technol..

[42]  W. DZIDA,et al.  User-Perceived Quality of Interactive Systems , 1978, IEEE Transactions on Software Engineering.

[43]  Jacques Leplat,et al.  Evaluation of Different Modalities of Verbalization in a Sorting Task , 1983, Int. J. Man Mach. Stud..