Personas in Heuristic Evaluation: An Exploratory Study

Research problem: No study has explored how incorporating personas into heuristic evaluation of products, namely websites, affects the kinds of findings reported and the recommendations presented by usability evaluators. Research questions: (1) Do findings resulting from heuristic evaluations of a website without the use of personas differ from findings resulting from heuristic evaluations of the same website with the use of personas? (2) Do findings from persona-based heuristic evaluations in which evaluators develop their own personas differ from findings from persona-based heuristic evaluations in which evaluators are given personas? (3) If findings and recommendations are different, how do they differ? (4) How does the use of personas affect the evaluators' confidence in the findings of a heuristic evaluation? Literature review: First, previous research of heuristic evaluation has concluded that although heuristic evaluation is inexpensive and does not require advance planning, it has several shortcomings, including its too-intense focus on minor issues and its inability to capture all usability issues. Second, data-driven personas, which have long been a resource in user-centered design, have been suggested as a way to improve or enhance heuristic evaluation, and several studies suggest that usability professionals are indeed using personas in their evaluations. However, no empirical study has assessed heuristic evaluations that include personas. Methodology: In this exploratory study involving three sections of an advanced technical writing course, groups of evaluators conducted a heuristic evaluation of a website. Each section was randomly assigned a different condition with which they would conduct the heuristic evaluation: (a) a traditional heuristic evaluation, (b) a persona-led heuristic evaluation in which the personas were given to the evaluators, or (c) a persona-led heuristic evaluation in which the evaluators themselves created their own personas. Each group wrote a report identifying the major problems with the website and provided recommendations to solve the identified problems. The evaluators completed pretesting demographic surveys and posttesting confidence surveys. Results and discussion: This exploratory study found few detectable differences in the findings reported by groups that used personas in heuristic evaluation and groups that did not use personas. The groups that used personas were more likely to report findings related to navigation than the groups that did not use personas, while the groups that did not use personas were more likely to report findings related to design than the groups that used personas. The groups that created their own personas were more likely than the other groups to include complex issues in their reports and include language that directly references users and user needs. All groups were confident in their findings.