Mode of questionnaire administration can have serious effects on data quality.

BACKGROUND One of the main primary data collection instruments in social, health and epidemiological research is the survey questionnaire. Modes of data collection by questionnaire differ in several ways, including the method of contacting respondents, the medium of delivering the questionnaire to respondents, and the administration of the questions. These are likely to have different effects on the quality of the data collected. METHODS This paper is based on a narrative review of systematic and non-systematic searches of the literature on the effects of mode of questionnaire administration on data quality. RESULTS Within different modes of questionnaire administration, there were many documented potential, biasing influences on the responses obtained. These were greatest between different types of mode (e.g. self-administered versus interview modes), rather than within modes. It can be difficult to separate out the effects of the different influences, at different levels. CONCLUSIONS The biasing effects of mode of questionnaire administration has important implications for research methodology, the validity of the results of research, and for the soundness of public policy developed from evidence using questionnaire-based research. All users of questionnaires need to be aware of these potential effects on their data.

[1]  C. Scott.  Research on Mail Surveys , 1961 .

[2]  S L Warner,et al.  Randomized response: a survey technique for eliminating evasive answer bias. , 1965, Journal of the American Statistical Association.

[3]  Chava Nachmias,et al.  Research Methods in the Social Sciences , 1976 .

[4]  J. Siemiatycki.  A comparison of mail, telephone, and home interview strategies for household health surveys. , 1979, American journal of public health.

[5]  N. Hatton.  ASKING QUESTIONS , 1979, The Medical journal of Australia.

[6]  Robert M. Groves,et al.  Surveys by Telephone: A National Comparison with Personal Interviews , 1980 .

[7]  John Dowling,et al.  Interview vs. paper-and-pencil administration of the McGill pain questionnaire , 1981, PAIN.

[8]  Norman M. Bradburn,et al.  Chapter 8 – Response Effects1 , 1983 .

[9]  S. Presser,et al.  Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context , 1996 .

[10]  James D. Wright,et al.  Handbook of Survey Research. , 1985 .

[11]  H. Genn.  Health Surveys in Practice and in Potential (Book). , 1986 .

[12]  D. Brambilla,et al.  A comparison of responses to mailed questionnaires and telephone interviews in a mixed mode health survey. , 1987, American journal of epidemiology.

[13]  J. Krosnick,et al.  AN EVALUATION OF A COGNITIVE THEORY OF RESPONSE-ORDER EFFECTS IN SURVEY MEASUREMENT , 1987 .

[14]  Roger Tourangeau,et al.  Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. , 1987 .

[15]  L W Chambers,et al.  Sensitivity to Change and the Effect of Mode of Administration on Health Status Measurement , 1987, Medical care.

[16]  D. Dooley.  Social Research Methods , 1990 .

[17]  R. Groves,et al.  Telephone Survey Methodology. , 1990 .

[18]  Leslie F. Clark,et al.  RATING SCALES NUMERIC VALUES MAY CHANGE THE MEANING OF SCALE LABELS , 1991 .

[19]  Hans-Jürgen Hippler,et al.  CONFIDENTIALITY ASSURANCES IN SURVEYS: REASSURANCE OR THREAT? , 1992 .

[20]  D. Leeuw,et al.  Data Quality in Mail, Telephone and Face to Face Surveys. , 1992 .

[21]  Michael R. Weeks,et al.  Computer-assisted survey information collection: a review of CASIC methods and their implications for survey operations: , 1992 .

[22]  J. E. Brazier,et al.  Validating the SF-36 health survey questionnaire: new outcome measure for primary care. , 1992, BMJ.

[23]  Agreement between interview data and a self-administered questionnaire on dietary supplement use. , 1994, European journal of clinical nutrition.

[24]  C. McHorney,et al.  Comparisons of the Costs and Quality of Norms for the SF-36 Health Survey Collected by Mail Versus Telephone Interview: Results From a National Survey , 1994, Medical care.

[25]  J. Santelli,et al.  AIDS and behavioural risk factors in women in inner city Baltimore: a comparison of telephone and face to face surveys. , 1994, Journal of epidemiology and community health.

[26]  Hans-Jürgen Hippler,et al.  SUBSEQUENT QUESTIONS MAY INFLUENCE ANSWERS TO PRECEDING QUESTIONS IN MAIL SURVEYS , 1995 .

[27]  Ian MacKay,et al.  ASKING THE QUESTIONS , 1995, Engineered to Speak.

[28]  H A Guess,et al.  Does the mode of questionnaire administration affect the reporting of urinary symptoms? , 1995, Urology.

[29]  G P Samsa,et al.  Are health-related quality-of-life measures affected by the mode of administration? , 1996, Journal of clinical epidemiology.

[30]  Seymour Sudman,et al.  Thinking about Answers: The Application of Cognitive Process to Survey Methodology , 1996 .

[31]  Tom W. Smith,et al.  ASKING SENSITIVE QUESTIONS THE IMPACT OF DATA COLLECTION MODE, QUESTION FORMAT, AND QUESTION CONTEXT , 1996 .

[32]  Ann Bowling,et al.  Research Methods in Health: Investigating Health and Health Services , 1997 .

[33]  W M Tierney,et al.  Assessing inner-city patients' hospital experiences. A controlled trial of telephone interviews versus mailed surveys. , 1997, Medical care.

[34]  R W Sanson-Fisher,et al.  An examination of self- and telephone-administered modes of administration for the Australian SF-36. , 1998, Journal of clinical epidemiology.

[35]  A. Bowling.  Research Methods in Health , 1998 .

[36]  S. Presser,et al.  Data collection mode and social desirability bias in self-reported religious attendance : Church attendance in the United States , 1998 .

[37]  S. M. Rogers,et al.  Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. , 1998, Science.

[38]  Bärbel Knäuper,et al.  The Impact of Age and Education on Response Order Effects in Attitude Measurement , 1999 .

[39]  F. J. Fowler,et al.  Comparing telephone and mail responses to the CAHPS survey instrument. Consumer Assessment of Health Plans Study. , 1999, Medical care.

[40]  A Bowling,et al.  Short Form 36 (SF-36) Health Survey questionnaire: which normative data should be used? Comparisons between the norms provided by the Omnibus Survey in Britain, the Health Survey for England and the Oxford Healthy Life Survey. , 1999, Journal of public health medicine.

[41]  Epidemiology deserves better questionnaires. , 1999, International journal of epidemiology.

[42]  D. Dillman.  Mail and internet surveys , 1999 .

[43]  Seymour Sudman,et al.  Cognition, aging, and self-reports , 1999 .

[44]  R A Lyons,et al.  SF-36 scores vary by method of administration: implications for study design. , 1999, Journal of public health medicine.

[45]  Denise C. Park,et al.  Cognitive Aging: A Primer , 1999 .

[46]  M. Carey,et al.  Self-Administered Questionnaires versus Face-to-Face Interviews in Assessing Sexual Behavior in Young Women , 2000, Archives of sexual behavior.

[47]  M. Dewey,et al.  Assessing research outcomes by postal questionnaire with telephone follow-up. TOTAL Study Group. Trial of Occupational Therapy and Leisure. , 2000, International journal of epidemiology.

[48]  Denise C. Park,et al.  The basic mechanisms accounting for age-related decline in cognitive function. , 2000 .

[49]  R. Pruchno,et al.  Interview Modality , 2000 .

[50]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[51]  R. Rusek,et al.  A comparison of the reliability of self-reported drug use and sexual behaviors using computer-assisted versus face-to-face interviewing. , 2000, AIDS education and prevention : official publication of the International Society for AIDS Education.

[52]  The Psychology of Survey Response: Preface , 2000 .

[53]  A. Kriska,et al.  Self-administered questionnaire compared with interview to assess past-year physical activity. , 2000, Medicine and science in sports and exercise.

[54]  M. Clarke,et al.  Methods to influence response to postal questionnaires , 2001 .

[55]  G. Locke,et al.  Prospective comparison of endoscopy patient satisfaction surveys: e-mail versus standard mail versus telephone , 2001, American Journal of Gastroenterology.

[56]  X. Badia,et al.  Effect of Mode of Administration on I–PSS Scores in a Large BPH Patient Population , 2001, European Urology.

[57]  Anne M Johnson,et al.  Effect of computer-assisted self-interviews on reporting of sexual HIV risk behaviours in a general population sample: a methodological experiment , 2001, AIDS.

[58]  E. Perez-stable,et al.  The effect of data collection mode on smoking attitudes and behavior in young African American and Latina women. Face-to-face interview versus self-administered questionnaires. , 2001, Evaluation review.

[59]  P. Ravaud,et al.  Impact of reminders and method of questionnaire distribution on patient response to mail-back satisfaction survey. , 2001, Journal of clinical epidemiology.

[60]  E McColl,et al.  Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. , 2001, Health technology assessment.

[61]  D Banister,et al.  A multidimensional model of the quality of life in older age , 2002, Aging & mental health.

[62]  Christopher J. McCarthy,et al.  Paper-and-Pencil Or Online? , 2002, Assessment.

[63]  N. Bellamy,et al.  A comparative study of telephone versus onsite completion of the WOMAC 3.0 osteoarthritis index. , 2002, The Journal of rheumatology.

[64]  D. Acar,et al.  Effect of administration mode (patient vs physician) and patient's educational level on the Turkish version of the International Prostate Symptom Score , 2002, International journal of urology : official journal of the Japanese Urological Association.

[65]  Bias in patient assessments of general practice: general practice assessment survey scores in surgery and postal responders. , 2003, The British journal of general practice : the journal of the Royal College of General Practitioners.

[66]  L. Midanik,et al.  Telephone versus in-person interviews for alcohol use: results of the 2000 National Alcohol Survey. , 2003, Drug and alcohol dependence.

[67]  A. Erol,et al.  Mode of administration of international prostate symptom score in patients with lower urinary tract symptoms: physician vs self , 2004, Prostate Cancer and Prostatic Diseases.

[68]  A. Glaser,et al.  Influence of proxy respondents and mode of administration on health status assessment following central nervous system tumours in childhood , 2004, Quality of Life Research.

[69]  T. Peters,et al.  Assessing mental health in primary care research using standardized scales: can it be carried out over the telephone? , 2004, Psychological Medicine.

[70]  Robyn Attewell,et al.  A comparison of an electronic version of the SF-36 General Health Questionnaire to the standard paper version , 2002, Quality of Life Research.

[71]  A. Wu,et al.  The effect of mode of administration on Medical Outcomes Study health ratings and EuroQol scores in AIDS , 2004, Quality of Life Research.

[72]  N. Schwarz,et al.  Thinking About Answers: The Application of Cognitive Processes to Survey Methodology , 1995, Quality of Life Research.

[73]  Sanjaya Mishra,et al.  Research methods in the social sciences , 2005 .

[74]  D. Dillman.  Mail and internet surveys: The tailored design method, 2nd ed. , 2007 .