Cognitive interviewing as tool for enhancing the accuracy of the interpretation of quantitative findings

This paper contrasts findings from a quantitative survey with those from a cognitive interviewing follow-up investigation on a subset of the same respondents. The data were gathered as part of a larger study to explore measurement error across three modes of data collection, but this paper focuses on the question format experiments rather than the mode effects part of the larger study. Three examples are presented which demonstrate how cognitive interviewing can cast new light on quantitative results by increasing the accuracy of the inferences made. These include instances where: (1) quantitative indicators of poor respondent behaviour (e.g., acquiescence bias on agree/disagree questions) are over-estimates, (2) similar quantitative response distributions across satisfaction and behavioural questions (from a fully-labelled versus end-labelled experiment) imply similar respondent satisficing behaviour, but cognitive interviews show that different response processes are at work and (3) unlikely quantitative findings (from an experiment comparing 3 vs. 7 or 8 response options) could easily be dismissed as due to chance but were instead the result of unforeseen respondent difficulties. The paper concludes with a discussion of the value of using a cognitive interviewing follow-up study as a tool in the interpretation of ambiguous quantitative findings.