Everyday concepts and classification errors: judgments of disability and residence

This article examines two neglected sources of misinterpretations of survey questions. Respondents may misunderstand the questions because the survey uses an everyday term in a technical way that differs from the everyday sense (and respondents fail to recognize the difference); in addition, respondents may have trouble applying the concept to borderline cases, situations that do not map neatly onto whichever sense of the concept they are trying to use. We call these two problems misalignment of the concepts and imperfect fit between concept and instance. We examined two everyday concepts – residence and disability – that figure prominently in surveys. Our initial experiment gave respondents definitions for residence and disability and asked them to classify vignettes describing concrete instances. We constructed one definition that reflected the everyday concept (the everyday definition) and a second one that departed from it (the technical definition). The vignettes varied in how well the instance depicted matched the requirements of the two definitions (for example, some vignettes clearly met the requirements of one definition but clearly did not meet the requirements of the other). Participants who got the technical definition for residence seemed able to keep it distinct from the everyday definition but those who got the technical definition for disability seemed to fall back on their everyday concept. In addition, participants were better at classifying vignettes that closely matched the definition (central instances) than ones that did not match it so well ( peripheral instances). In our second experiment, we tried to encourage participants to pay more attention to the definition of the concepts by giving the concepts unfamiliar labels (e.g., calling a residence an enumeration unit). Although the use of unfamiliar terminology did increase the proportion of respondents who consulted the definition of the concept as they judged the vignettes, there was strong evidence (especially for disability) that participants still relied on their everyday sense of the concepts. Respondents were again more accurate in classifying central than peripheral instances. People have difficulty using concepts in unfamiliar ways and, when they have to, they often make errors. Even with concepts used in their everyday senses, people have trouble classifying borderline instances, those that only partially match the definition of the concept. Category membership is graded, rather than all or none, and people have trouble dealing with cases that are near the concept’s boundary. We suspect that both misalignment and imperfect fit often produce substantial measurement error in surveys.

[1]  F. Conrad,et al.  Use and non-use of clarification features in web surveys , 2006 .

[2]  W. Dijkstra,et al.  Response latencies and (para)linguistic expressions as indicators of response error , 2004 .

[3]  Frederick G. Conrad,et al.  Misunderstanding Standardized Language in Research Interviews , 2004 .

[4]  Robert L. Goldstone,et al.  Concepts and Categorization , 2003 .

[5]  G. Murphy,et al.  Paper has been my ruin: Conceptual relations of polysemous senses , 2002 .

[6]  Nancy S. Kim,et al.  Clinical psychologists' theory-based representations of mental disorders predict their diagnostic reasoning and memory. , 2002, Journal of experimental psychology. General.

[7]  G. Murphy,et al.  The Representation of Polysemous Words , 2001 .

[8]  G. Murphy,et al.  Feature Distribution and Background Knowledge in Category Learning , 2000, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[9]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[10]  F. Conrad,et al.  Clarifying question meaning in a household telephone survey. , 2000, Public opinion quarterly.

[11]  M. Couper.  A REVIEW OF ISSUES AND APPROACHES , 2000 .

[12]  G. S. Wunderlich,et al.  Conceptual Issues in the Measurement of Work Disability , 2000 .

[13]  Brian H. Ross,et al.  Food for Thought: Cross-Classification and Category Organization in a Complex Real-World Domain , 1999, Cognitive Psychology.

[14]  Frederick G. Conrad,et al.  Does Conversational Interviewing Reduce Survey Measurement Error , 1997 .

[15]  J. N. Bassili,et al.  RESPONSE LATENCY AS A SIGNAL TO QUESTION PROBLEMS IN SURVEY RESEARCH , 1996 .

[16]  E. Gerber,et al.  “WHO LIVES HERE?”: THE USE OF VIGNETTES IN HOUSEHOLD ROSTER RESEARCH , 1996 .

[17]  L. Hirschfeld.  Mapping the mind: Is the acquisition of social categories based on domain-specific competence or on knowledge transfer? , 1994 .

[18]  R. Nosofsky,et al.  Rule-plus-exception model of classification learning. , 1994, Psychological review.

[19]  N. Pennington,et al.  Explaining the evidence: Tests of the Story Model for juror decision making. , 1992 .

[20]  D L Medin,et al.  Concepts and conceptual structure. , 1989, The American psychologist.

[21]  Paul Kay,et al.  Prototype semantics: The English Word Lie , 1981 .

[22]  W. Belson,et al.  The design and understanding of survey questions , 1982 .

[23]  E. Rosch,et al.  Family resemblances: Studies in the internal structure of categories , 1975, Cognitive Psychology.

[24]  Lance J. Rips,et al.  Structure and process in semantic memory: A featural model for semantic decisions. , 1974 .