Instant data analysis: conducting usability evaluations in a day

When designing a usability evaluation, key decisions must be made regarding methods and techniques for data collection and analysis. Although there is a strong body of research within human-computer interaction regarding the appropriate choices of data collection methods and techniques much less research has been conducted examining and comparing methods and techniques for analyzing the collected data. This paper presents a data analysis technique which allows usability evaluations to be conducted, analyzed and documented in a day; Instant Data Analysis (IDA). The use of this technique is exemplified through a usability evaluation of a software product for a large hospital for which traditional video data analysis and Instant Data Analysis were applied independently through a controlled experiment. Among our key findings, the experiment revealed that in only 10% of the time required to do the video data analysis, Instant Data Analysis identified 85% of the critical usability problems in the system being evaluated. At the same time, the noise of unique usability problems usually characterizing video data analysis was significantly reduced.

[1]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[2]  David R. Smith,et al.  Analyzing and communicating usability data: now that you have the data what do you do? a CHI'94 workshop , 1995, SGCH.

[3]  Morten Hertzum,et al.  The evaluator effect in usability tests , 1998, CHI Conference Summary.

[4]  Clare-Marie Karat,et al.  Comparison of empirical testing and walkthrough methods in user interface evaluation , 1992, CHI.

[5]  Jan Stage,et al.  New techniques for usability evaluation of mobile systems , 2004, Int. J. Hum. Comput. Stud..

[6]  Jakob Nielsen,et al.  Guerrilla HCI: using discount usability engineering to penetrate the intimidation barrier , 1994 .

[7]  Randolph G. Bias,et al.  Cost-justifying usability , 2005 .

[8]  Jeffrey Rubin,et al.  Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests , 1994 .

[9]  Jesper Kjeldskov,et al.  Creating Realistic Laboratory Settings: Comparative Studies of Three Think-Aloud Usability Evaluations of a Mobile System , 2003, INTERACT.

[10]  Robert A. Virzi,et al.  Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough? , 1992 .

[11]  Wayne D. Gray,et al.  Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods , 1998, Hum. Comput. Interact..

[12]  Gavriel Salvendy,et al.  Designing and using human-computer interfaces and knowledge based systems , 1989 .

[13]  Connor Graham,et al.  Mobile evaluation: what the data and the metadata told us , 2003 .

[14]  Jakob Nielsen,et al.  Usability engineering at a discount , 1989 .

[15]  P NayakNandini,et al.  Analyzing and communicating usability data , 1995 .

[16]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[17]  A. Strauss,et al.  Grounded Theory in Practice , 1997 .