Testing the Validity of Gender Ideology Items by Implementing Probing Questions in Web Surveys

This article examines the use of probing techniques in web surveys to identify validity problems of items. Conventional cognitive interviewing is usually based on small sample sizes and thus precludes quantifying the findings in a meaningful way or testing small or special subpopulations characterized by their response behavior. This article investigates probing in web surveys as a supplementary way to look at item validity. Data come from a web survey in which respondents were asked to give reasons for selecting a response category for a closed question. The web study was conducted in Germany, with respondents drawn from online panels (n = 1,023). The usefulness of the proposed approach is shown by revealing validity problems with a gender ideology item.

[1]  G. Willis,et al.  Cognitive Interviewing: A Tool for Improving Questionnaire Design , 2004 .

[2]  Willem E. Saris,et al.  Design, Evaluation, and Analysis of Questionnaires for Survey Research: Saris/Design , 2007 .

[3]  J. Macinnes.  Analysing patriarchy capitalism and women's employment in Europe , 1998 .

[4]  Frederick G. Conrad,et al.  Sources of Error in Cognitive Interviews , 2009 .

[5]  Jennifer Hess,et al.  THE ROLE OF RESPONDENT DEBRIEFING QUESTIONS IN QUESTIONNAIRE DEVELOPMENT , 1998 .

[6]  Jolene D. Smyth,et al.  Open-Ended Questions in Web Surveys Can Increasing the Size of Answer Boxes and Providing Extra Verbal Instructions Improve Response Quality? , 2009 .

[7]  G. Willis,et al.  Evaluation of Cognitive Interviewing Techniques: Do the Results Generalize to the Field? , 1997 .

[8]  Mirta Galesic,et al.  Dropouts on the Web: Influence of changes in respondents' interest and perceived burden during the Web survey , 2004 .

[9]  Lars Kaczmirek,et al.  Asking Probing Questions in Web Surveys , 2012 .

[10]  J. Blasius,et al.  Assessing Data Quality and Construct Comparabilityin Cross-National Surveys , 2006 .

[11]  Michael Braun.  Using Egalitarian Items to Measure Men’s and Women’s Family Roles , 2008 .

[12]  Frederick G. Conrad,et al.  Data Quality in Cognitive Interviews: The Case of Verbal Reports , 2004 .

[13]  Leah Melani Christian,et al.  The Influence of Topic Interest and Interactive Probing on Responses to Open-Ended Questions in Web Surveys , 2009 .

[14]  Willem E. Saris,et al.  Design, Evaluation, and Analysis of Questionnaires for Survey Research , 2007 .

[15]  RANDOM PROBES OF GSS QUESTIONS , 1989 .

[16]  Martyn Denscombe,et al.  The Length of Responses to Open-Ended Questions , 2008 .

[17]  G. Willis,et al.  Research Synthesis: The Practice of Cognitive Interviewing , 2007 .

[18]  E. Nichols,et al.  Respondent Debriefings Conducted by Experts: A Technique for Questionnaire Evaluation , 2009 .

[19]  R. Vandenberg,et al.  A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research , 2000 .

[20]  Jon A. Krosnick,et al.  Research Synthesis AAPOR Report on Online Panels , 2010 .

[21]  Theresa J. DeMaio,et al.  Cognitive interviewing techniques: In the lab and in the field. , 1996 .

[22]  H. Schuman,et al.  The random probe: a technique for evaluating the validity of closed questions. , 1966, American sociological review.