Item comparability in cross-national surveys: results from asking probing questions in cross-national web surveys about attitudes towards civil disobedience

This article focuses on assessing item comparability in cross-national surveys by asking probing questions in Web surveys. The “civil disobedience” item from the “rights in a democracy” scale of the International Social Survey Program (ISSP) serves as a substantive case study. Identical Web surveys were fielded in Canada (English-speaking), Denmark, Germany, Hungary, Spain, and the U.S. A category-selection and a comprehension probe, respectively, were incorporated into the Web surveys after the closed-ended “civil disobedience” item. Responses to the category selection-probe reveal that notably in Germany, Hungary, and Spain the detachment of politicians from the people and their lack of responsiveness is deplored. Responses to the comprehension probe show that mainly in the U.S. and Canada violence and/or destruction are associated with civil disobedience. These results suggest reasons for the peculiar statistical results found for the “civil disobedience” item in the ISSP study. On the whole, Web probing proves to be a valuable tool for identifying interpretation differences and potential bias in cross-national survey research.

[1]  G. Willis,et al.  Cognitive Interviewing: A Tool for Improving Questionnaire Design , 2004 .

[2]  Deirdre Lawrence,et al.  Methods for Translating an English-Language Survey Questionnaire on Tobacco Use into Mandarin, Cantonese, Korean, and Vietnamese , 2007 .

[3]  Norbert Schwarz,et al.  Cognition and Communication: Judgmental Biases, Research Methods, and the Logic of Conversation , 2014 .

[4]  Jihyun Lee.  Conducting Cognitive Interviews in Cross-National Settings , 2014, Assessment.

[5]  H. Coffé,et al.  A Gender Gap in Citizenship Norms? The Importance of Political, Civil and Social Rights and Responsibilities , 2008 .

[6]  Theresa J. DeMaio,et al.  Cognitive interviewing techniques: In the lab and in the field. , 1996 .

[7]  David R. Williams,et al.  Comparing Cognitive Interviewing and Psychometric Methods to Evaluate a Racial/Ethnic Discrimination Scale , 2011, Field methods.

[8]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[9]  G. Willis,et al.  Research Synthesis: The Practice of Cognitive Interviewing , 2007 .

[10]  S. Bowler,et al.  Trust in Government: The United States in Comparative Perspective , 2008 .

[11]  Patricia L. Goerman,et al.  Bilingual questionnaire evaluation and development through mixed pretesting methods: the case of the U.S. census Nonresponse Followup instrument , 2010 .

[12]  Gregory M Dominick,et al.  Using Cognitive Interviewing and Behavioral Coding to Determine Measurement Equivalence across Linguistic and Cultural Groups , 2011, Field methods.

[13]  P. Dreier.  The United States in Comparative Perspective , 2007 .

[14]  Sue Ellen Hansen,et al.  Designing Questionnaires for Multipopulation Research , 2010 .

[15]  R. Abelson,et al.  Social Information Processing and Survey Methodology. , 1987 .

[16]  D. Collins,et al.  Identifying sources of error in cross-national questionnaires: Application of an error source typology to cognitive interview data , 2011 .

[17]  Deirdre Lawrence,et al.  Translation of a tobacco survey into Spanish and Asian languages: the Tobacco Use Supplement to the Current Population Survey. , 2008, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.

[18]  J. Vermunt,et al.  Exploring the Response Process of Culturally Differing Survey Respondents with a Response Style , 2013 .

[19]  H. Schuman,et al.  The random probe: a technique for evaluating the validity of closed questions. , 1966, American sociological review.

[20]  F. Strack,et al.  Thinking, Judging, and Communicating: A Process Account of Context Effects in Attitude Surveys , 1987 .

[21]  Seymour Sudman,et al.  Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research , 1995 .

[22]  J. Padilla,et al.  Design and Analysis of Cognitive Interviews for Comparative Multinational Testing , 2011, Field methods.

[23]  Lars Kaczmirek,et al.  Testing the Validity of Gender Ideology Items by Implementing Probing Questions in Web Surveys , 2013 .

[24]  Lars Kaczmirek,et al.  Asking Probing Questions in Web Surveys , 2012 .

[25]  J. Harkness.  Survey Methods in Multinational, Multiregional, and Multicultural Contexts , 2010 .

[26]  J. Citrin,et al.  Trust in government. , 1999 .

[27]  Patricia L. Goerman,et al.  A preferred approach for the cognitive testing of translated materials: testing the source version as a basis for comparison , 2010 .