Estimation of the effects of measurement characteristics on the quality of survey questions

When designing questionnaires, many choices have to be made. Because the consequences of these choices for the quality of the questions are largely unknown, it has often been said that designing a questionnaire is an art. To make it a more scientific activity we need to know more about the consequences of these choices. In order to further such an approach we have: 1. made an inventory of the choices to be made when designing survey questions and created a code book to transform these question characteristics into the independent variables for explaining quality of survey questions; 2. assembled a large set of studies that use Multi-trait Multi-method experiments to estimate the reliability and validity of questions 3. carried out a meta-analysis that relates these question characteristics to the reliability and validity estimates of the questions. On the basis of the results of these efforts we have constructed a database. This data base contains at present 1023 measurement instruments based on 87 experiments conducted on random samples from sometimes regional but mostly national samples of 300 to 2000 respondents. The database contains information on studies of reliability and validity of survey questions formulated in three different languages: English, German and Dutch. The purpose of this study was to generate cross national generalizations of the findings published so far drawn from national studies. This analysis provides a quantitative estimate of the effects of the different choices on the reliability, validity and the method effects.

[1]  Willem E. Saris,et al.  Variation in response functions : a source of measurement error in attitude research , 1988 .

[2]  Willem E. Saris,et al.  Different Explanations for Correlated Disturbance Terms in MTMM Studies , 2003 .

[3]  A. C. Scherpenzeel.  A Question of Quality. Evaluating survey questions by multitrait-multimethod studies , 1995 .

[4]  F. M. Andrews.  Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach , 1984 .

[5]  E. Diener,et al.  Handbook of Multimethod Measurement in Psychology , 2005 .

[6]  D. Dillman.  Mail and internet surveys , 1999 .

[7]  J. L. Esposito,et al.  Evaluating Survey Data: Making the Transition from Pretesting to Quality Assessment , 1997 .

[8]  Willem E. Saris,et al.  Design, Evaluation, and Analysis of Questionnaires for Survey Research , 2007 .

[9]  J. van der Zouwen,et al.  An assessment of the difficulty of questions used in the ISSP-questionnaires, the clarity of their wording, and the comparability of the responses , 2000 .

[10]  Willem E. Saris,et al.  8. A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design , 2004 .

[11]  Hans-Dieter Klingemann.  The left-right self-placement question in face to face and telephone surveys , 1997 .

[12]  Werner Wothke,et al.  Models for Multitratit-Multimethod Matrix Analysis , 1996 .

[13]  Willem E. Saris,et al.  Fit of Different Models for Multitrait-Multimethod Experiments , 2002 .

[14]  Arthur C. Graesser,et al.  QUAID: A questionnaire evaluation aid for survey methodologists , 2000, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[15]  N. Schwarz,et al.  Thinking About Answers: The Application of Cognitive Processes to Survey Methodology , 1995, Quality of Life Research.

[16]  Stanley Presser,et al.  SURVEY PRETESTING: DO DIFFERENT METHODS PRODUCE DIFFERENT RESULTS? , 1994 .

[17]  Willem E. Saris,et al.  Operationalization of Social Science Concepts by Intuition , 2004 .

[18]  Seymour Sudman,et al.  Measurement errors in surveys , 1993 .

[19]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[20]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[21]  Willem E. Saris,et al.  Testing Nested Additive, Multiplicative, and General Multitrait-Multimethod Models , 2000 .

[22]  Annette Scherpenzeel,et al.  The Validity and Reliability of Survey Questions , 1997 .

[23]  Irmtraud N. Gallhofer,et al.  Formulierung und Klassifikation von Fragen. , 2000 .

[24]  Esposito Jl,et al.  Methodologies for evaluating survey questions: some lessons from the redesign of the Current Population Survey. , 1993 .

[25]  Seymour Sudman,et al.  Thinking about Answers: The Application of Cognitive Process to Survey Methodology , 1996 .

[26]  R. Bagozzi,et al.  Multitrait-Multimethod Matrices in Consumer Research , 1991 .

[27]  W. Belson,et al.  The design and understanding of survey questions , 1982 .

[28]  D. Alwin,et al.  Feeling Thermometers Versus 7-Point Scales , 1997 .

[29]  J. Conley.  Asking questions: A practical guide to questionnaire design , 1983 .

[30]  Eli P. Cox,et al.  The Optimal Number of Response Alternatives for a Scale: A Review , 1980 .

[31]  H. Marsh,et al.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: A Comparison of Alternative Models , 1991 .

[32]  Annette Scherpenzeel,et al.  Validity and Reliability of Subjective Social Indicators: The effect of different measures of association , 1998 .

[33]  Jaak Billiet,et al.  Het survey-interview onderzocht. Effecten van het ontwerp en gebruik van vragenlijsten op de kwaliteit van de antwoorden , 1984 .

[34]  Peter Wiemer-Hastings,et al.  THE GOLD STANDARD OF QUESTION QUALITY ON SURVEYS: EXPERTS, COMPUTER TOOLS, , 2001 .

[35]  D. A. Kenny,et al.  Analysis of the multitrait^multimethod matrix by confirmatory factor analysis , 1992 .