The usability inspection performance of work-domain experts: An empirical study

It is a challenge for usability experts to perform usability inspections of interactive systems that are tailored to work-domains of which these experts have little knowledge. To counter this, usability inspections with work-domain experts have been explored, but little empirical research has been reported on these experts' performance as evaluators. The present study compared the performance of work-domain experts and usability experts with respect to validity and thoroughness. The work-domain experts were characterized by high computer experience and low system experience. The usability experts were recruited from different ICT companies. The usability inspection method applied was group-based expert walkthrough; a method particularly developed to support non-usability experts as evaluators. The criterion for performance comparison was established through user tests. Fifteen work-domain experts and 12 usability experts participated in the study. The work-domain experts generated equally valid but less thorough usability inspection results than did the usability experts. This finding implies that work-domain experts may be used as evaluators in usability inspections without compromising validity. Moreover, the usability inspection performance of nominal groups of evaluators was explored. It was found that nominal groups of work-domain experts produced results of similar quality as did nominal groups of usability experts, given that group size is disregarded. This finding may be used as basis for hypotheses in future studies on the usability inspection performance of nominal groups of work-domain experts.

[1]  Audris Mockus,et al.  Variability and Reproducibility in Software Engineering: A Study of Four Companies that Developed the Same System , 2009, IEEE Transactions on Software Engineering.

[2]  Michael J. Muller,et al.  Validating an extension to participatory heuristic evaluation: quality of work and quality of work life , 1995, CHI 95 Conference Companion.

[3]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[4]  Michael E. Atwood,et al.  What is gained and lost when using evaluation methods other than empirical testing , 1993 .

[5]  Wayne D. Gray,et al.  Repairing Damaged Merchandise: A Rejoinder , 1998, Hum. Comput. Interact..

[6]  Robert A. Virzi.  Usability Inspection Methods , 1997 .

[7]  Ann Blandford,et al.  People and Computers XV - Interaction without Frontiers : Joint Proceedings of HCI 2001 and IHM 2001 , 2001 .

[8]  Gilbert Cockton,et al.  Understanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation , 2001, BCS HCI/IHM.

[9]  Wayne D. Gray,et al.  Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods , 1998, Hum. Comput. Interact..

[10]  Andy Field,et al.  Discovering statistics using SPSS, 2nd ed. , 2005 .

[11]  Andrew Sears,et al.  Heuristic Walkthroughs: Finding the Problems Without the Noise , 1997, Int. J. Hum. Comput. Interact..

[12]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[13]  David Clark-Carter,et al.  Doing Quantitative Psychological Research: From Design To Report , 1997 .

[14]  Ebba Þóra Hvannberg,et al.  Analysis of strategies for improving and estimating the effectiveness of heuristic evaluation , 2004, NordiCHI '04.

[15]  Jan Gulliksen,et al.  Use and Usefulness of HCI Methods: Results from an Exploratory Study among Nordic HCI Practitioners , 2005, BCS HCI.

[16]  Tonya L. Smith-Jackson,et al.  Supporting novice usability practitioners with usability engineering tools , 2009, Int. J. Hum. Comput. Stud..

[17]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[18]  Jacqueline Brodie,et al.  Applying user testing data to UEM performance metrics , 2004, CHI EA '04.

[19]  T. Landauer,et al.  Handbook of Human-Computer Interaction , 1997 .

[20]  Bonnie E. John,et al.  Tracking the effectiveness of usability evaluation methods , 1997, Behav. Inf. Technol..

[21]  AsbjØRn F⊘lstad,et al.  Work-Domain Experts as Evaluators: Usability Inspection of Domain-Specific Work-Support Systems , 2007 .

[22]  E. Krahmer,et al.  Thinking about thinking aloud: a comparison of two verbal protocols for usability testing , 2004, IEEE Transactions on Professional Communication.

[23]  Jan Gulliksen,et al.  Integrating work environment considerations into usability evaluation methods - the ADA approach , 2003, Interact. Comput..

[24]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[25]  Ebba Þóra Hvannberg,et al.  Heuristic evaluation: Comparing ways of finding and reporting usability problems , 2007, Interact. Comput..

[26]  Giorgio Venturi,et al.  Survey on the UCD integration in the industry , 2004, NordiCHI '04.

[27]  Robert C. Williges,et al.  Criteria For Evaluating Usability Evaluation Methods , 2003, Int. J. Hum. Comput. Interact..

[28]  Gilbert Cockton,et al.  Inspection-based evaluations , 2002 .

[29]  Yvonne Rogers,et al.  PETRA: Participatory Evaluation Through Redesign and Analysis , 1995, Interact. Comput..

[30]  Gavriel Salvendy,et al.  Effect of evaluators' cognitive style on heuristic evaluation: Field dependent and field independent evaluators , 2009, Int. J. Hum. Comput. Stud..

[31]  Michael J. Muller,et al.  Participatory design: the third space in HCI , 2002 .

[32]  Michael J. Muller,et al.  Methods & tools: participatory heuristic evaluation , 1998, INTR.

[33]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[34]  Jan Gulliksen,et al.  People and Computers XIX - The Bigger Picture: Proceedings of HCI 2005 (BCS Conference) , 2005 .

[35]  Cathleen Wharton,et al.  Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces , 1990, CHI '90.

[36]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[37]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[38]  Kasper Hornbæk,et al.  Cooperative usability testing: complementing usability tests with user-supported interpretation sessions , 2005, CHI Extended Abstracts.

[39]  Finn Kensing,et al.  PD: structure in the toolbox , 1993, CACM.

[40]  Gilbert Cockton,et al.  Sale must end: should discount methods be cleared off HCI's shelves? , 2002, INTR.

[41]  Kim Halskov,et al.  Methods & tools: context: an active choice in usability work , 1998, INTR.

[42]  Heather Desurvire,et al.  EMPIRICISM VERSUS JUDGEMENT: COMPARING USER INTERFACE EVALUATION METHODS ON A NEW TELEPHONE-BASED INTERFACE , 1991, SGCH.

[43]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[44]  Richard Bentley,et al.  Situated evaluation for cooperative systems , 1994, CSCW '94.

[45]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[46]  Bente Anda,et al.  Assessing Software System Maintainability using Structural Measures and Expert Assessments , 2007, 2007 IEEE International Conference on Software Maintenance.

[47]  Jan Stage,et al.  Improving the Interplay between Usability Evaluation and User Interface Design , 2004 .

[48]  Gitte Lindgaard,et al.  Notions of thoroughness, efficiency, and validity: Are they valid in HCI practice? , 2006 .

[49]  Roy Rada,et al.  Interacting WITH Computers , 1989, Interact. Comput..

[50]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[51]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[52]  Bente Anda,et al.  A multiple-case study of software effort estimation based on use case points , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..