On the reliability of usability testing

Six professional usability testing teams conducted a usability test on an early prototype of a dialog box. Altogether, they identified 36 usability problems. No problem was detected by every team, 2 were found by five teams, 4 by four teams, 7 by three teams, 7 by two teams, and 18 problems were identified by one team only. There was more agreement among teams in this study compared to a previous study [1] and there was more agreement among the teams on severe vs. minor problems. Implications for the cooperation between usability testers and their clients are discussed.

[1]  Lars Schmidt,et al.  Comparative evaluation of usability tests , 1999, CHI Extended Abstracts.