Playing your cards right: getting the most from card sorting for navigation design

Card sorting is a knowledge-elicitation technique often used by information architects, interaction designers, and usability professionals to establish or assess the navigation hierarchy of a Web site. The items are typically menu entries or hyperlinks, while the groups are categories or headings. The process involves asking participants to sort items into meaningful groups. In open card sorts, the number and names of groups are decided by each participant, while in the closed card sorts, these factors are fixed by the researcher in advance. Analysis of card-sorting results range from simple counting of the number of times items were grouped together to the rather intimidating “monothetic agglomerative cluster analysis” (known simply as cluster analysis in most cases). Unfortunately, no single technique provides everything a researcher needs to know, especially if convincing evidence is needed to persuade colleagues or customers of the effectiveness of a proposed design. The evidence we need falls into three categories: Participants. Are these the right participants for our site? Are they all thinking about the items and their groupings in a similar way? Do they have a clear understanding of the card-sorting task itself? Items. Are the item names well-understood by participants? Are there alternatives that should be considered—perhaps terms users are more familiar with? Groups. For closed card sorts, have we chosen the right number of groups and names for each? For open sorts, are participants largely in agreement about the number of groups needed? How well do participants feel the items fit into their groups? Happily, the answer to this last question—how well participants feel the items fit into their groups—can also help us with many of the other issues listed. Coupled with a few data-collection guidelines and alternative presentations of results, we can collect fairly comprehensive evidence about what will and will not work in our navigation hierarchies. So let’s examine this last question in some more detail: How well do participants feel the items fit into their groups? It is possible to argue that this question is redundant, that the items must fit into their groups relatively well in any given set of results, because that is how the participant decided to group them. However, practical experience says otherwise. Consider the following example that I use as a practice sorting exercise when teaching: Participants are given the names of 14 wines and asked to sort them into three groups (full-bodied reds, dry whites, and sparkling). Participants are instructed to omit any items they feel do not really belong to any of the groups. The cluster analysis dendogram shown in figure 1 is a fairly typical set of results for 12 participants. The dendogram shows the three groups, connected in the characteristic tree-like structure that gives this form of presentation its name. The vertical connections between branches indicate the strength of the relationship between items, with stronger relationships to the right and weaker to the left. So for example, the relationship between Riesling and White Zinfandel is the strongest in this dendogram, meaning that those two items appeared in the same group more frequently than any other pair of items. The relationship between Beaujolais and Claret is only slightly less strong, while the weakest relationship between any single item and its groups is Pinot Grigio. But for wine lovers, there is something fishy about this result. If you remember, participants were asked to group the wines into three categories, one of which was full-bodied reds. While Beaujolais is a red wine, it certainly cannot be described as full-bodied (there is also a problem with White Zinfandel that I am not going to deal with here—it was a nasty trick played on participants that will p eo p le whudson@acm.org