An assessment of the difficulty of questions used in the ISSP-questionnaires, the clarity of their wording, and the comparability of the responses

The rationale of cross-national survey research rests on the assumptions (1) that the answers given by respondents are sufficiently informative about the behaviours and opinions asked for in the questionnaires, and (2) that the answers of respondents participating in surveys, conducted in different countries, are sufficiently comparable, despite differences in the languages used. It is investigated whether these assumptions are justified for a large selection of surveys conducted within the context of the International Social Survey Program (ISSP). A conceptual model based on methodological research after response effects guides this evaluation of questionnaires. The model asserts that response quality depends on the difficulty and clarity of the questions, on the length of the questionnaire and on the presence or absence of biasing factors. Questions from ISSP-questionnaires are evaluated by using ten criteria relating to question difficulty and three indicators concerning the clarity of the questions. The comparability of responses to these questions is evaluated by looking at six different aspects, like 'equivalent wording of the question' and 'similarity of the objects to be evaluated'. This evaluation shows that some questions asked in the ISSP-surveys are rather difficult while others are not very clear. The comparability of the responses is also hampered by the fact that different modes of questionnaire administration are used in the participating countries and that the attitude objects to be evaluated by respondents may differ between questionnaires.