Impact of the mode of data collection on the quality of survey questions in social sciences

Esta tesis estudia el impacto que el metodo de recoleccion de datos en encuestas tiene sobre la calidad de las respuestas, definida como el producto de la fiabilidad y la validez. Utilizando datos de Holanda sobre temas diversos (utilizacion de los medios de comunicacion, confianza social y politica, satisfaccion, orientaciones politicas, auto-ubicacion en la escala izquierdaderecha, actitudes hacia la inmigracion), se muestra que la calidad es similar en una encuesta cara-a-cara asistida con ordenador y utilizando tarjetas (la Encuesta Social Europea) y una encuesta online basada en una muestra probabilistica (el panel LISS). Esto se cumple tanto para los indicadores simples, como para indicadores complejos. Los resultados sugieren que las relaciones estandardizadas entre variables son comparables entre los dos metodos de recoleccion. Al contrario, las entrevistas telefonicas producen diferencias de calidad. Para conceptos complejos, la equivalencia de las mediciones tambien esta garantizada: las medias y las relaciones no estandarizadas son comparables en las entrevistas cara-a-cara y online.

[1]  Jeffrey A. Miller,et al.  The Invariance of the Self-Concept Construct across White and Hispanic Student Populations , 1997 .

[2]  Annette Scherpenzeel,et al.  The Evaluation of Measurement Instruments By Meta-Analysis of Multitrait-Multimethod Studies , 1993 .

[3]  D. Alwin.  Approaches to the Interpretation of Relationships in the Multitrait-Multimethod Matrix , 1973 .

[4]  Tom W. Smith,et al.  ASKING SENSITIVE QUESTIONS THE IMPACT OF DATA COLLECTION MODE, QUESTION FORMAT, AND QUESTION CONTEXT , 1996 .

[5]  F. Kreuter,et al.  Social Desirability Bias in CATI, IVR, and Web Surveys The Effects of Mode and Question Sensitivity , 2008 .

[6]  E. Roberts Caroline,et al.  Mixing modes of data collection in surveys: A methodological review , 2007 .

[7]  Seppo Laaksonen,et al.  Methods for achieving equivalence of samples in cross-national surveys: the European Social Survey experience , 2004 .

[8]  Jagdip Singh,et al.  Measurement Issues in Cross-National Research , 1995 .

[9]  Robert K. Merton,et al.  Sociology Today: Problems and Prospects , 1962 .

[10]  A. E. Maxwell,et al.  Factor Analysis as a Statistical Method. , 1964 .

[11]  Annette Scherpenzeel,et al.  The Validity and Reliability of Survey Questions , 1997 .

[12]  Dirk Heerwegh,et al.  Mode Differences Between Face-to-Face and Web Surveys: An Experimental Investigation of Data Quality and Social Desirability Effects , 2009 .

[13]  J. Krosnick,et al.  Comparing questions with agree/ disagree response options to questions with item-specific response options , 2010 .

[14]  E. Uslaner.  The Moral Foundations of Trust , 2002 .

[15]  K. Jöreskog.  Statistical analysis of sets of congeneric tests , 1971 .

[16]  K. Hergenrather,et al.  Collecting behavioural data using the world wide web: considerations for researchers , 2003, Journal of epidemiology and community health.

[17]  Willem E. Saris,et al.  Mixed mode designs: finding the balance between nonresponse bias and mode effects , 2005 .

[18]  L. Ross,et al.  Perseverance of Social Theories: The Role of Explanation in the Persistence of Discredited Information , 1980 .

[19]  M. Coenders,et al.  Nationalistic Attitudes and Ethnic Exclusionism in a Comparative Perspective: An Empirical Study of Attitudes Toward the Country and Ethnic Immigrants in 22 countries , 2001 .

[20]  F. M. Andrews.  Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach , 1984 .

[21]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[22]  Martin E. P. Seligman,et al.  The attributional Style Questionnaire , 1982, Cognitive Therapy and Research.

[23]  Arthur G. Bedeian,et al.  The Measurement Equivalence of Web-Based and Paper-and-Pencil Measures of Transformational Leadership , 2006 .

[24]  Melanie Revilla,et al.  Quality in Unimode and Mixed-Mode designs: A Multitrait-Multimethod approach , 2010 .

[25]  A. Mayda.  Who Is Against Immigration? A Cross-Country Investigation of Individual Attitudes toward Immigrants , 2004, The Review of Economics and Statistics.

[26]  Willem E. Saris,et al.  Evaluation of Measurement Instruments Using a Structural Modeling Approach , 2011 .

[27]  Willem E. Saris,et al.  Fit of Different Models for Multitrait-Multimethod Experiments , 2002 .

[28]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[29]  Edith de Leeuw,et al.  Methodological Issues in Survey Research : a Historical Review , 1999 .

[30]  Melanie C. Green,et al.  Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias , 2003 .

[31]  Laurent Lesnard.  ELIPSS: A New Mobile Web Panel for Social Scientists , 2011 .

[32]  K. Vos,et al.  The representativeness of LISS, an online probability panel , 2009 .

[33]  Floyd J. Fowler,et al.  Survey Research Methods , 1984 .

[34]  Duane F. Alwin.  Margins of Error: A Study of Reliability in Survey Measurement , 2007 .

[35]  Willem E. Saris,et al.  Memory effects in MTMM studies , 1995 .

[36]  Angela L. Duckworth,et al.  Self-discipline gives girls the edge : Gender in self-discipline, grades, and achievement test scores , 2006 .

[37]  Geert Molenberghs,et al.  A Method for Evaluating Mode Effects in Mixed-mode Surveys , 2010 .

[38]  Sunghee Lee.  Propensity score adjustment as a weighting scheme for volunteer panel web surveys , 2006 .

[39]  Robert L. Linn,et al.  Path analysis: Psychological examples. , 1970 .

[40]  Don A. Dillman,et al.  INFLUENCE OF AN INVITATION TO ANSWER BY TELEPHONE ON RESPONSE TO CENSUS QUESTIONNAIRES , 1994 .

[41]  Michael D. Kaplowitz,et al.  A Comparison of Web and Mail Survey Response Rates , 2004 .

[42]  Willem E. Saris,et al.  8. A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design , 2004 .

[43]  W. Meredith.  Measurement invariance, factor analysis and factorial invariance , 1993 .

[44]  Don A. Dillman,et al.  Survey Mode as a Source of Instability in Responses across Surveys , 2005 .

[45]  M. Browne,et al.  The decomposition of multitrait-multimethod matrices. , 1984, The British journal of mathematical and statistical psychology.

[46]  Scott Fricker,et al.  An Experimental Comparison of Web and Telephone Surveys , 2005 .

[47]  Willem E. Saris,et al.  A Comparison of the Quality of Questions in a Face-to-face and a Web Survey , 2013 .

[48]  Jon A. Krosnick,et al.  The Reliability of Survey Attitude Measurement , 1991 .

[49]  Willem E. Saris,et al.  Design, Evaluation, and Analysis of Questionnaires for Survey Research , 2007 .

[50]  Vera Toepoel,et al.  Effects of Design in Web Surveys Comparing Trained and Fresh Respondents , 2008 .

[51]  Robert D. Tortora,et al.  Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet , 2009 .

[52]  Filip Lievens,et al.  Measurement Equivalence of Paper‐and‐Pencil and Internet Organisational Surveys: A Large Scale Examination in 16 Countries , 2009 .

[53]  E. Leeuw,et al.  To mix or not to mix data collection modes in surveys. , 2005 .

[54]  F. W. Vondracek,et al.  Factorial Invariance of Career Indecision Dimensions across Junior High and High School Males and Females. , 1988 .

[55]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[56]  Dale Trusheim,et al.  The Case against the SAT , 1988 .

[57]  M. Smyth.  Cognition In Action , 1987 .

[58]  R. Gerard,et al.  The logic of the sciences and the humanities , 1948 .

[59]  R. Groves,et al.  Telephone Survey Methodology. , 1990 .

[60]  D. Brambilla,et al.  A comparison of responses to mailed questionnaires and telephone interviews in a mixed mode health survey. , 1987, American journal of epidemiology.

[61]  J. Krosnick,et al.  AN EVALUATION OF A COGNITIVE THEORY OF RESPONSE-ORDER EFFECTS IN SURVEY MEASUREMENT , 1987 .

[62]  Vera Toepoel,et al.  Design of Web Questionnaires: A Test for Number of Items Per Screen , 2005 .

[63]  K. Bollen,et al.  Pearson's R and Coarsely Categorized Measures , 1981 .

[64]  Geert Loosveldt,et al.  Face-to-Face versus Web Surveying in a High-Internet-Coverage Population Differences in Response Quality , 2008 .