The influence of the design of web survey questionnaires on the quality of responses

he first objective of this article is to propose a conceptual framework of the effects of on-line questionnaire design on the quality of collected responses. Secondly, we present the results of an experiment where different protocols have been tested and compared in a randomised design using the basis of several quality indexes. Starting from some previous categorizations, and from the main factors identified in the literature, we first propose an initial global framework of the questionnaire and question characteristics in a web survey, divided into five groups of factors. Our framework was built to follow the response process successive stages of the contact between the respondent and the questionnaire itself. Then, because it has been studied in the survey methodology literature in a very restricted way, the concept of `response quality' is discussed and extended with some more `qualitative' criteria that could be helpful for researchers and practitioners, in order to obtain a deeper assessment of the survey output. As an experiment, on the basis of the factors chosen as major characteristics of the questionnaire design, eight versions of a questionnaire related to young people's consumption patterns were created. The links to these on-line questionnaires were sent in November 2005 to a target of 10,000 young people. The article finally presents the results of our study and discusses the conclusions. Very interesting results come to light; especially regarding the influence of length, interaction and question wording dimensions on response quality. We discuss the effects of Web-questionnaire design characteristics on the quality of data.

[1]  Roger Tourangeau,et al.  What They See Is What We Get , 2004 .

[2]  Robert M. Groves,et al.  UNDERSTANDING THE DECISION TO PARTICIPATE IN A SURVEY , 1992 .

[3]  William Foddy,et al.  Constructing Questions for Interviews and Questionnaires: Frontmatter , 1993 .

[4]  H. Cooper,et al.  A Quantitative Review of Research Design Effects on Response Rates to Questionnaires , 1983 .

[5]  Don A. Dillman,et al.  The Influence of Graphical and Symbolic Language Manipulations on Responses to Self-Administered Questions , 2004 .

[6]  Tracy L. Tuten,et al.  Classifying Response Behaviors in Web-based Surveys , 2006, J. Comput. Mediat. Commun..

[7]  C. Proctor,et al.  Response behaviour in the survey-interview , 1983 .

[8]  Don A. Dillman,et al.  Chapter 10 – Mail and Other Self-Administered Questionnaires1 , 1983 .

[9]  Robert M. Groves,et al.  Nonresponse in Household Interview Surveys: Groves/Nonresponse , 1998 .

[10]  Michael D. Kaplowitz,et al.  A Comparison of Web and Mail Survey Response Rates , 2004 .

[11]  M. Galesic.  Effects of questionnaire length on response rates : Review of findings and guidelines for future research , 2002 .

[12]  W. Belson,et al.  The design and understanding of survey questions , 1982 .

[13]  H. Schuman.  Improving Interview Method and Questionnaire Design: Response Effects to Threatening Questions in Survey Research.By Norman M. Bradburn, Seymour Sudman, Edward Blair, William Locander, Carrie Miles, Eleanor Singer, and Carol Stocking. San Francisco: Jossey-Bass, 1979. 214 pp. $13.95 , 1980 .

[14]  B. Duffy,et al.  Comparing Data from Online and Face-to-face Surveys , 2005 .

[15]  Scott Fricker,et al.  An Experimental Comparison of Web and Telephone Surveys , 2005 .

[16]  T. Tyler,et al.  Promised Incentives in Media Research: A Look at Data Quality, Sample Representativeness, and Response Rate , 1984 .

[17]  Cihan Cobanoglu,et al.  A Comparison of Mail, Fax and Web-Based Survey Methods , 2001 .

[18]  Anja S. Göritz,et al.  The impact of material incentives on response quantity, response quality, sample composition, survey outcome, and cost in online access panels , 2004 .

[19]  Terry Macpherson,et al.  An Empirical Evaluation of Three Web Survey Design Principles , 2005 .

[20]  Niels Schillewaert,et al.  Comparing response distributions of offline and online data collection methods , 2005 .

[21]  Jean Moscarola,et al.  Protocoles d'enquête et efficacité des sondages par Internet , 2004 .

[22]  Vasja Vehovar,et al.  Survey Design Features Influencing Response Rates in Web Surveys , 2002 .

[23]  Lee Sproull,et al.  Using Electronic Mail for Data Collection in Organizational Research , 1986 .

[24]  Russel L. Thompson,et al.  A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys , 2000 .

[25]  Edward G. Sargis,et al.  Social psychological research and the Internet : the promise and peril of a new methodological frontier , 2005 .

[26]  D. Dillman.  Mail and internet surveys: The tailored design method, 2nd ed. , 2007 .

[27]  Norbert Schwarz,et al.  Statistical Analysis of Choice Experiments and Surveys , 2005 .

[28]  Jennifer Preece,et al.  Electronic Survey Methodology: A Case Study in Reaching Hard-to-Involve Internet Users , 2003, Int. J. Hum. Comput. Interact..

[29]  Martin Wetzels,et al.  An Assessment of Equivalence Between Online and Mail Surveys in Service Research , 2006 .

[30]  M. Elliott,et al.  Conducting Research Surveys Via E-Mail and The Web , 2001 .

[31]  Geert Loosveldt,et al.  An Evaluation of the Effect of Response Formats on Data Quality in Web Surveys , 2002 .

[32]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[33]  Gerald Albaum,et al.  A Comparison of Response Characteristics from Web and Telephone Surveys , 2004 .

[34]  Frederick G. Conrad,et al.  Interactive Features of Web Surveys. , 2007 .

[35]  Robert D. Tortora,et al.  Principles for Constructing Web Surveys , 1998 .

[36]  J. Bachman,et al.  Effects of Questionnaire Length on Response Quality , 1981 .

[37]  Karen Cortell Reisman.  "The art of asking questions". , 2008, Texas dental journal.

[38]  J. Grémy.  Les expériences françaises sur la formulation des questions d'enquête. Résultats d'un premier inventaire , 1987 .

[39]  R. Calantone,et al.  Do Certified Mail Third-Wave Follow-ups Really Boost Response Rates and Quality? , 2005 .

[40]  Mick P. Couper,et al.  New Technologies and Survey Data Collection: Challenges and Opportunities , 2002 .

[41]  F. Conrad,et al.  Interactive Feedback Can Improve the Quality of Responses in Web Surveys , 2005 .

[42]  E. Singer,et al.  Leverage-saliency theory of survey participation: description and an illustration. , 2000, Public opinion quarterly.

[43]  Ulf-Dietrich Reips,et al.  Context Effects in Web Surveys , 2002 .

[44]  Martin Wetzels,et al.  Response Rate and Response Quality of Internet-Based Surveys: An Experimental Study , 2004 .

[45]  S. Baron,et al.  ONLINE SURVEYS IN MARKETING RESEARCH: PROS AND CONS , 2002 .

[46]  The Effect of Question Tone and Form on Responses to Open-Ended Questions: Further Data , 1999 .

[47]  S. Day,et al.  Survey Questions: Handcrafting the Standardized Questionnaire. , 1987 .

[48]  M. Traugott,et al.  Web survey design and administration. , 2001, Public opinion quarterly.

[49]  D. Stewart,et al.  From consumer response to active consumer: Measuring the effectiveness of interactive media , 2002 .

[50]  Samuel J. Best,et al.  Internet Data Collection (Quantitative Applications in the Social Sciences) , 2004 .

[51]  Tracy L. Tuten,et al.  Internet Surveys and Data Quality : A Review , 2002 .

[52]  F. Conrad,et al.  Spacing, Position, and Order Interpretive Heuristics for Visual Features of Survey Questions , 2004 .