DO RESPONDENTS REALLY “MARK ALL THAT APPLY” ON SELF-ADMINISTERED QUESTIONS?

An experiment was conducted to assess the effect of using "mark all that apply" question instructions on survey re- porting as part of the field test for the Second Follow-up of the National Education Longitudinal Study of 1988 Eighth Graders (NELS:88). Mark-all-that-apply instructions were compared with instructions asking respondents to indicate "yes" or "no" to each response option on responses to three items dispersed throughout the questionnaire and consisting of different topics and numbers of response options. For the three items, signifi- cantly fewer response options were selected with the mark-all- that-apply instructions than with the yes/no instructions, but be- cause external validity criteria were not available, overreporting to the yes/no instructions cannot be ruled out. Instruction- dependent primacy effects, predicted under the hypothesis that respondents would engage in more superficial processing when given the mark-all-that-apply instructions, were not found. In an attempt to reduce response burden, designers of self-admin- istered survey questionnaires often use the technique of asking respon- dents to examine a list and mark, circle, or check all the items on the list that apply. The mark-all-that-apply instructions have been used in many circumstances. One application is in questions asking about a set of respondent behaviors. For example, respondents might be asked, "Which of the following have you done in order to finance your child's education after high school? (Mark all that apply.)" or "To KENNETH A. RASINSKI is a National Opinion Research Center (NORC) senior survey methodologist, DAVID MINGAY is a survey methodologist at NORC, and NORMAN M. BRADBURN is NORC senior vice president for research and Tiffany and Margaret Blake