The Effect of the Question on Survey Responses: A Review

(Read before the ROYAL STATISTICAL SOCIETY at a meeting organized by the SOCIAL STATISTICS SECTION Oll Wednesday, September 30th, 1981, Professor G. HOINVILLE in the Chair) SUMMARY The paper reviews the effects of the precise wording, form and placement of questions in a survey questionnaire on the responses obtained. Topics discussed include: randomized response; respondent instructions; feedback and commitment; question length; the treatment of "don't knows"; open and closed questions; the use of balanced questions; acquiescence; the offer of a middle alternative; the order of alternatives; and question order and context effects. THE survey literature abounds with examples demonstrating that survey responses may be sensitive to the precise wording, format and placement of the questions asked. A useful start to examining these effects is to classify questions according to the type of information sought. A widely-used distinction is that between factual and opinion questions. Questions like "What was your regular hourly rate of pay on this job as of September 30?" clearly fall in the former category, while questions like "As you know, many older people share a home with their grown children. Do you think this is generally a good idea or a bad idea?" clearly fall in the latter. However, not all survey questions can be classified as either factual or opinion ones: other types of question include questions testing respondents' knowledge, questions asking for reasons, hypothetical questions and preference questions. One further type of question, widely used in survey practice, deserves special comment. These questions, which have a factual component overlaid with an evaluation, may be termed judgement or perceptual questions. Examples are: "Do you have street (highway) noise in this neighbourhood?" and "Would you say your health now is excellent, good, fair or poor?" In many cases the intent of such questions is to obtain factual information, but the approach adopted seeks respondents' evaluations of the facts rather than their measurement according to objective criteria. The use of perceptual questions for this purpose probably results from the questionnaire designer's decision that he could not ask sufficient questions or take the measurements necessary to determine the information objectively; hence he has respondents make the assessments for him. The often low levels of correlation found between perceptions and facts make this use of perceptual questions, although widespread, a dubious one. A different use of perceptual questions is indeed to obtain respondents' perceptions of their situations; in this case the questions are similar to opinion questions. For present purposes, it will be sufficient to divide questions into factual and non-factual ones (including as factual questions those perceptual questions seeking to ascertain factual information). An important difference between these two types of question is that with factual questions there are individual true values for the information sought which can-at least in

[1]  Stanley Presser,et al.  Context Effects on Survey Responses to Questions About Abortion , 1981 .

[2]  Peter V. Miller,et al.  Research on Interviewing Techniques , 1981 .

[3]  Robert W. Oldendick,et al.  Experiments in filtering political opinions , 1980 .

[4]  Robert W. Oldendick,et al.  Pseudo-Opinions on Public Affairs , 1980 .

[5]  S. Presser,et al.  The Measurement of a Middle Position in Attitude Surveys , 1980 .

[6]  D. Holt,et al.  The Effects of Offering a Middle Response Option with Opinion Questions , 1980 .

[7]  Stanley Presser,et al.  Public Opinion and Public Ignorance: The Fine Line Between Attitudes and Nonattitudes , 1980, American Journal of Sociology.

[8]  Duane F. Alwin,et al.  MEASUREMENT MODELS FOR RESPONSE ERRORS IN SURVEYS: ISSUES AND APPLICATIONS , 1980 .

[9]  Seymour Sudman,et al.  Improving Interview Method and Questionnaire Design. , 1982 .

[10]  Stanley Presser,et al.  The Open and Closed Question , 1979 .

[11]  M. Traugott,et al.  Response Validity in Surveysof Voting Behavior , 1979 .

[12]  N. Rothwell,et al.  Studies of Census Mail Questionnaires , 1979 .

[13]  B. M. Hedges.  Question Wording Effects: Presenting One or Both Sides of a Case , 1979 .

[14]  Stanley Presser,et al.  The Assessment of "No Opinion" in Attitude Surveys , 1979 .

[15]  Graham Kalton,et al.  Experiments in Wording Opinion Questions , 1978 .

[16]  C. Turner,et al.  Fallible indicators of the subjective state of the nation. , 1978 .

[17]  G. S. Bonham,et al.  Randomized response technique in a National Survey. , 1978, Journal of the American Statistical Association.

[18]  A. Laurent,et al.  A summary of studies of interviewing methodology. , 1978, Vital and health statistics. Series 2, Data evaluation and methods research.

[19]  Charles F. Cannel.  A Summary of Studies of Interviewing Methodology. Vital and Health Statistics: Series 2, Data Evaluation and Methods Research; No. 69. DHEW Publication No. (HRA) 77-1343. , 1977 .

[20]  S. Presser,et al.  Question Wording as an Independent Variable in Survey Analysis , 1977 .

[21]  J. Abernathy,et al.  Purposive concealment of death in household surveys in Misamis Oriental Province. , 1976, Population studies.

[22]  N. Bradburn,et al.  An Investigation of Interview Method, Threat and Response Distortion , 1976 .

[23]  John P. Burton,et al.  The Effect of Question Form on Gathering Income Data by Telephone , 1976 .

[24]  Michael S. Goodstadt,et al.  The Randomized Response Technique: A Test on Drug Use , 1975 .

[25]  Graham J. G. Upton,et al.  The Determination of the Optimum Position on a Ballot Paper , 1975 .

[26]  Graham J. G. Upton,et al.  Biases in Local Government Elections Due to Position on the Ballot Paper , 1974 .

[27]  The Importance of Positional Voting Bias in British Elections , 1974 .

[28]  Seymour Sudman,et al.  Effects of Time and Memory Factors on Response in Surveys , 1973 .

[29]  J. Peretz,et al.  A Quality Check on the 1966 Ten Per Cent Sample Census of England and Wales. , 1973 .

[30]  W S Cash,et al.  Optimum recall period for reporting persons injured in motor vehicle accidents. , 1972, Vital and health statistics. Series 2, Data evaluation and methods research.

[31]  W. Belson,et al.  The effects of reversing the order of presentation of verbal rating scales in survey interviews , 1969 .

[32]  W. Belson,et al.  The effects of reversing the presentation order of verbal rating scales , 1966 .

[33]  S L Warner,et al.  Randomized response: a survey technique for eliminating evasive answer bias. , 1965, Journal of the American Statistical Association.

[34]  Angus Campbell,et al.  The American voter , 1960 .

[35]  A. Barton,et al.  Asking the Embarrassing Question , 1958 .

[36]  Stanley L. Payne,et al.  The Art of Asking Questions , 1951 .

[37]  Hugh J. Parry,et al.  Validity of Responses to Survey Questions , 1950 .

[38]  Gertrude Bancroft,et al.  Recent Experience with Problems of Labor Force Measurement , 1946 .

[39]  Q. Mcnemar,et al.  Opinion-attitude methodology. , 1946, Psychological bulletin.

[40]  W. Edwards Deming,et al.  On Errors in Surveys , 1944 .

[41]  Hadley Cantril,et al.  II. THE WORDING OF QUESTIONS , 1944 .

[42]  S. Fiske,et al.  The Handbook of Social Psychology , 1935 .