Website usability evaluation: reliability of questionnaires in the South African context

The usability of interactive Web interfaces is a factor of increasing importance and intensive research has been done towards measuring the usability of interactive Web interfaces. Usability evaluation includes both objective measures such as time, errors, completion and completion rate, and the subjective measured as user satisfaction. Questionnaires are often used as a tool in measuring user satisfaction with interactive websites. In this paper we investigated the reliability of post-test questionnaires as an evaluation tool in the South African context. The study was done at a South African, multi-cultural, open-distance learning university. We conducted two usability studies that included formal laboratory testing, eye tracking, and post-test questionnaires. The first study investigated the usability of an assignment submission Learning Management System while the second study focused on the usability of the university information portal. In contrast to expectation and existing literature on the correlation between the objective measures (time, errors and completions time) and the subjective (satisfaction), the findings based on the post-test questionnaire data was not consistent with the findings based on the formal laboratory testing and the eye tracking data. Participants reported a positive experience with the use of the websites but the formal laboratory testing and eye tracking data provided evidence that several participants had difficulty in using the websites. The contrast between the positive user satisfaction measured by the post-test questionnaire and the usability problems experienced by the same users in the first study was confirmed by the results of the second study. Having validated the objective results this discrepancy raises questions about the measuring of user satisfaction and hence the reliability of using post-test questionnaires in interactive website evaluation. The contribution of this paper is to highlight the problems with using post-test questionnaires in a South African context. The reliability of post-test questionnaires in usability testing has implications for e- learning, e-government and other websites that provide services to a diverse population.