THE GOLD STANDARD OF QUESTION QUALITY ON SURVEYS: EXPERTS, COMPUTER TOOLS,

38152 Abstract QUAID (Question Understanding Aid) is a computer tool that assists survey methodologists who want to improve the wording, syntax, and semantics of questions on surveys. QUAID produces a list of potential problems with each question, including (1) unfamiliar technical term, (2) vague or imprecise relative term, (3) vague or ambiguous noun-phrase, (4) complex syntax, (5) working memory overload, and (6) misleading or incorrect presupposition. We assessed the incidence of these problems on a corpus of 11 surveys provided by the US Census Bureau. We are currently facing the challenge of assessing the validity of QUAID's critique of questions. The collection of multiple methods and measures is a lofty virtue, but this approach offers no principled foundation for rectifying discrepancies in performance measures. This presentation addresses the challenges of performance evaluation when there is no defensible gold standard for question quality.

[1]  James F. Allen.  Natural language understanding , 1987, Bejnamin/Cummings series in computer science.

[2]  Peter M. Wiemer-Hastings,et al.  DP: A Detector for Presuppositions in survey questions , 2000, ANLP.

[3]  J. Jobe,et al.  Cognition and survey measurement: History and overview , 1991 .

[4]  Frederick G. Conrad,et al.  Does Conversational Interviewing Reduce Survey Measurement Error , 1997 .

[5]  Judith T. Lessler,et al.  A coding system for appraising questionnaires. , 1996 .

[6]  G. Willis,et al.  The use of verbal report methods in the development and testing of survey questionnaires , 1991 .

[7]  M. Sirken,et al.  Laboratory-based research on the Cognitive Aspects of Survey Methodology: the goals and methods of the National Center for Health Statistics study. , 1985, The Milbank Memorial Fund quarterly. Health and society.

[8]  Seymour Sudman,et al.  Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research , 1995 .

[9]  Charles F. Cannell,et al.  Using behavioral coding to identify cognitive problems with survey questions. , 1996 .

[10]  R. Groves,et al.  Survey Errors and Survey Costs. , 1991 .

[11]  Wendy G. Lehnert,et al.  Information extraction: What have we learned? , 1997 .

[12]  Eric R. Ziegel,et al.  Nonsampling Error in Surveys , 1992 .

[13]  Arthur C. Graesser,et al.  QUAID: A questionnaire evaluation aid for survey methodologists , 2000, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.