Effects of Questionnaire Length on Response Quality

A response tendency resulting from the length of a group-administered questionnaire instrument is described. Respondents answering items that are included in large sets toward the later parts of a long questionnaire are more likely to give identical answers to most or all of the items, compared with those responding to items in smaller sets or in shorter questionnaires. While means and intercorrelations among items within the same set are affected by this "straight-line" response pattern, intercorrelations between items from different sets are much less affected by it. These investigations are based on comparisons between a long questionnaire, administered to 1,050 high school seniors in nine high schools across the nation in 1978, and five shorter questionnaires administered to large national samples of high school seniors. A. Regula Herzog is Study Director and Jerald G. Bachman is Program Director at the Survey Research Center, The University of Michigan, Ann Arbor, Michigan 48104. The authors would like to thank Pamela KIttel for competent research assistance, Frank Andrews, Patrick O'Malley, Willard Rodgers, and Howard Schuman for helpful comments on a draft of this paper. This research was supported by Grant No. NIE-G-780036 of the National Institute of Education. Public Opinion Quarterly Vol. 45:549-559 ? 1981 by The Trustees of Columbia University Published by Elsevier North-Holland, Inc. 0033-362X/81/0045-549/$2.50 This content downloaded from 207.46.13.145 on Wed, 27 Apr 2016 05:55:25 UTC All use subject to http://about.jstor.org/terms 550 HERZOG AND BACHMAN Little empirical evidence is available for either contention, particularly with regard to response quality. A study by Clancy and Wachsler (1971) on personal interviews shows no substantial increase in agreement bias or willingness to report socially undesirable behaviors when questions are asked at the end of a 25-minute telephone interview rather than at the beginning. Sudman and Bradburn (1974) report little effect of the position of a question in the survey instrument in their systematic review of response effects in surveys, but also acknowledged that only a few relevant studies are available. On the other hand, two studies of the responses to questionnaires that were administered in groups are more suggestive of a negative effect of length of instrument. Findings reported by Kraut et al. (1975) show an increasing use of the modal response category and decreasing use of the extreme response categories in later parts of a 168-item questionnaire. Johnson et al. (1974) observed fewer responses to an open-ended question when such a question was included at the end rather than at the beginning of an 18-item questionnaire. We recently had the occasion to examine some effects of questionnaire length on response quality as part of an ongoing study of high school seniors. The occasion was provided by the administration of a long questionnaire, which was a composite of five shorter questionnaire forms. The short questionnaires were designed so that the great majority of high school seniors would be able to complete them within a single 45-minute class period. In order to cover a broad range of topics despite the imposed time restriction, multiple questionnaire forms were used (with roughly 40 percent of the material common to all forms and 60 percent unique to each form). Since this design severely restricts correlational analyses in which items from one form are related to items from another form, a "long" form was developed combining most of the unique material from the five forms. It was recognized that the resulting long questionnaire, which required on the average over two hours to complete, would be much more demanding than the usual shorter forms, and that this would require both special efforts to motivate respondents and considerable caution in interpreting results. The efforts to motivate respondents included a five-dollar payment and released time from class (we judge the latter to be a positive factor for most students). The caution in interpreting results involved a considerable amount of comparison between long form data and the corresponding items from the shorter forms. On the whole, the comparisons of long and short forms revealed rather little evidence of systematic differences across a variety of means and correlations. We did, however, identify one particular effect of questionnaire length which appears to be limited to long sets This content downloaded from 207.46.13.145 on Wed, 27 Apr 2016 05:55:25 UTC All use subject to http://about.jstor.org/terms EFFECTS OF QUESTIONNAIRE LENGTH 551 of items using identical response scales. The effect manifests itself as an increased tendency to use an identical response category for all items in such a set, a form of responding that we label in the following paragraphs as "straight-line" responding. In other words, respondents are increasingly likely to show some form of position bias in later parts of the questionnaire. This paper documents the existence of this response pattern and examines its possible effects on substantive