Reducing Mode Effects in "Mark All That Apply" Questions. Princeton, NJ: Mathematica Policy Research, Inc.

Introduction Using data fi-om a mode effects study conducted as part of the 1993 National Survey of College Graduates, this paper looks at the extent to which mode effects in "Mark All That Apply" questions are minimized when such questions are reformatted to require a "yes or no" response after each response alternative. A "Mark All That Apply" format, upon reading the question, asks respondents to look over a list of response categories and mark only those categories that apply. Because it is viewed as an effective format for minimizing respondent burden, it is frequently used. One serious shortcoming, however, is its vulnerability to mode effects. More specifically, visually presented "Mark All That Apply" questions in self-administered instruments are prone to "primacy effects"-the tendency to select response alternatives from the top of a list, while orally presented (telephone-administered) "Mark All That Apply" questions are prone to "recency effects" or the tendency to mark items toward the bottom of the list. Items placed early in a list have a definite advantage in self-administered questionnaires. As discussed by Tourangeau (1984), respondents work through four stages of cognitive processing when responding to a question: (1) comprehension-interpreting the meaning of the question; (2) referral--retrieving relevant material from memory; (3) judgement--using the relevant material from memory to ~brmulate a response; and (4) reporting--responding in a manner consistent with prior answers or with regard to other factors such as social desirability. To give every question in an entire interview or questionnaire such careful and deliberate consideration would require more time and effort than most respondents are willing or able to provide. Consequently, respondents generally seek to minimize their burden by "weak satisficing" or selecting the first reasonable response ti-om the list of alternatives presented. (Schuman and Presser 1981, Krosnick and Alwin 1987, Krosnick 1991). This means initially listed response categories in a "Mark All That Apply" format are often more carefully considered than later ones, and thoughts relevant to those initial categories often influence how later response categories are perceived. The prevalence of recency effects in telephoneadministered questionnaires is influenced by the limitations of memory and the fact that respondents to telephone surveys tend to feel they have less control over the pace of the interview than do respondents to self-administered questionnaires. Since not prolonging the call seems to be an unspoken rule of telephone interviewing, telephone respondents are often less likely than respondents to a selfadministered questionnaire to stop and ponder a particular response category or ask to have categories reread. Similarly, with respect to memory, it appears most people cannot retain more than two or three response categories in memory at a time, again giving the last categories heard an advantage over response categories read earlier in a list. Cumulatively, both of these factors tend to favor making selections from among the final categories read (Schuman and Presser 1981, Tarnai and Dillman, Krosnick 1991, Krosnick and Alwin 1987, Swartz and Strack 1991). Sudman and Bradburn (1982) recognlized the shortcomings of "Mark All That Apply" questions more than a decade ago. They suggested data quality would improve if these questions were reformatted such that respondents were asked to indicate a negative or affirmative response to each individual response category, as opposed to only marking those that applied. While "more cumbersome," they argued it would encourage respondents to more thoroughly read and consider the response categories (Ibid, page 168). Interestingly, very little research has been conducted on this issue. The only directly relevant research we uncovered was a recent experiment conducted by Rasinski, Mingay and Bradburn (Rasinski et al. 1994). As part of a selfadministered questionnaire pretest, half the sample members were presented three questions formatted with "mark all that apply" instructions. The other half were presented the same three questions in a form that asked for a "yes or no" response after each item. Among the three questions, one had four response categories, one had twelve and one had twenty. They found the "yes~no'" format produced more "yes" responses, but it was not clear whether this reflected better reporting accuracy or overreporting. The two items with fewest response categories did not show primacy effects in the "yes~no" format, but primacy was still evident in the one item with 20 response categories. The authors, however, felt these results should be taken with some caution given the limited number of items being tested.