Are We Overlooking Some Usability Testing Methods? A Comparison of Lab, Beta, and Forum Tests

Abstract We compared the effectiveness of lab testing, beta testing, and forum testing at identifying software usability problems. Thirty participants were involved in the experiment, with ten participants in each of the three test conditions. The lab test involved participants performing prescribed scenarios with the software in a controlled lab environment, while human factors engineers recorded participant's problems. The beta test method had participants use the software in their own environment to perform their real world work and record their own problems. The forum test was similar to the beta test, except that the software was made available on a company-wide computer bulletin board and the participants selected themselves. Findings show that the beta test method was as effective as the lab test method in the number of problem types identified. The lab test uncovered a larger proportion of serious usability problems than did the beta test. The beta test method was the most cost-effective method. T...