An Empirical Validation Study of Popular Survey Methodologies for Sensitive Questions

When studying sensitive issues, including corruption, prejudice, and sexual behavior, researchers have increasingly relied upon indirect questioning techniques to mitigate such known problems of direct survey questions as underreporting and nonresponse. However, there have been surprisingly few empirical validation studies of these indirect techniques because the information required to verify the resulting estimates is often difficult to access. This article reports findings from the first comprehensive validation study of indirect methods. We estimate whether people voted for an anti-abortion referendum held during the 2011 Mississippi General Election using direct questioning and three popular indirect methods: list experiment, endorsement experiment, and randomized response. We then validate these estimates against the official election outcome. While direct questioning leads to significant underestimation of sensitive votes against the referendum, indirect survey techniques yield estimates much closer to the actual vote count, with endorsement experiment and randomized response yielding the least bias.

[1]  Jacob N. Shapiro,et al.  Relative Poverty, Perceived Violence, and Support for Militant Politics: Evidence from Pakistan* , 2016, Political Science Research and Methods.

[2]  Kosuke Imai,et al.  Design and Analysis of the Randomized Response Technique , 2015 .

[3]  Kenneth F. Greene,et al.  Using the Predicted Responses from List Experiments as Explanatory Variables in Regression Models , 2015, Political Analysis.

[4]  K. Imai,et al.  Comparing and Combining List and Endorsement Experiments: Evidence from Afghanistan , 2014 .

[5]  Leonardo Bursztyn,et al.  Identifying Ideology: Experimental Evidence on Anti-Americanism in Pakistan , 2014 .

[6]  Kosuke Imai,et al.  Explaining Support for Combatants during Wartime: A Survey Experiment in Afghanistan , 2013, American Political Science Review.

[7]  Ivar Krumpal,et al.  Estimating the prevalence of xenophobia and anti-Semitism in Germany: A comparison of randomized response and direct questioning. , 2012, Social science research.

[8]  Jacob N. Shapiro,et al.  Statistical Analysis of Endorsement Experiments: Measuring Support for Militant Groups in Pakistan , 2011, Political Analysis.

[9]  Kosuke Imai,et al.  Multivariate Regression Analysis for the Item Count Technique , 2011 .

[10]  Ben Jann,et al.  Sensitive Questions in Online Surveys: Experimental Results for the Randomized Response Technique (RRT) and the Unmatched Count Technique (UCT) , 2011 .

[11]  Daniel W. Gingerich.  Understanding Off-the-Books Politics: Conducting Inference on the Determinants of Sensitive Behavior with Randomized Response Surveys , 2010, Political Analysis.

[12]  J. Krosnick,et al.  Measuring Voter Turnout By Using The Randomized Response Technique Evidence Calling Into Question The Method’s Validity , 2010 .

[13]  Jon A. Krosnick,et al.  Social desirability bias in voter turnout reports Tests using the item count technique , 2010 .

[14]  Daniel Corstange,et al.  Sensitive Questions, Truthful Answers? Modeling the List Experiment with LISTIT , 2009, Political Analysis.

[15]  Jochen Musch,et al.  A randomized‐response investigation of the education effect in attitudes towards foreigners , 2009 .

[16]  M. G. Pittau,et al.  A weakly informative default prior distribution for logistic and other regression models , 2008, 0901.4011.

[17]  R. Tourangeau,et al.  Sensitive questions in surveys. , 2007, Psychological bulletin.

[18]  Peter G. M. van der Heijden,et al.  The logistic regression model with response variables subject to randomized response , 2007, Comput. Stat. Data Anal..

[19]  Takahiro Tsuchiya,et al.  A Study of the Properties of the Item Count Technique , 2007 .

[20]  Peter G. M. van der Heijden,et al.  Meta-Analysis of Randomized Response Research , 2005 .

[21]  M. Earleywine,et al.  Sexual risk behaviors and alcohol: Higher base rates revealed using the unmatched‐count technique , 2000 .

[22]  J. Hox,et al.  A Comparison of Randomized Response, Computer-Assisted Self-Interview, and Face-to-Face Direct Questioning , 2000 .

[23]  James H. Kuklinski,et al.  Racial Attitudes and the "New South" , 1997, The Journal of Politics.

[24]  Dan R. Dalton,et al.  USING THE UNMATCHED COUNT TECHNIQUE (UCT) TO ESTIMATE BASE RATES FOR SENSITIVE BEHAVIOR , 1994 .

[25]  Henk Elffers,et al.  On measuring tax evasion , 1992 .

[26]  D. O’Keefe.  Persuasion , 1990, The Handbook of Communication Skills.

[27]  M. Junger.  Discrepancies between police and self-report data for Dutch racial minorities , 1989 .

[28]  Henk Elffers,et al.  Exploring the limits of self-reports and reasoned action: An investigation of the psychology of tax evasion behavior. , 1988 .

[29]  Franziska Marquart,et al.  Communication and persuasion : central and peripheral routes to attitude change , 1988 .

[30]  J. Cacioppo,et al.  Central and Peripheral Routes to Advertising Effectiveness: The Moderating Role of Involvement , 1983 .

[31]  Thomas A. Buchman,et al.  Obtaining Responses To Sensitive Questions - Conventional Questionnaire Versus Randomized-Response Technique , 1982 .

[32]  Paul E. Tracy,et al.  The Validity of Randomized Response for Sensitive Measurements , 1981 .

[33]  S. Chaiken.  Heuristic versus systematic information processing and the use of source versus message cues in persuasion. , 1980 .

[34]  Charles W. Lamb,et al.  An Empirical Validation of the Randomized Response Technique , 1978 .

[35]  N. Bradburn,et al.  An Investigation of Interview Method, Threat and Response Distortion , 1976 .

[36]  S. Warner.  The Linear Randomized Response Model , 1971 .

[37]  W. R. Simmons,et al.  The Unrelated Question Randomized Response Model: Theoretical Framework , 1969 .

[38]  S L Warner,et al.  Randomized response: a survey technique for eliminating evasive answer bias. , 1965, Journal of the American Statistical Association.

[39]  Katrin Baumgartner,et al.  Persuasion Theory And Research , 2016 .

[40]  Leonardo Bursztyn,et al.  NBER WORKING PAPER SERIES A REVEALED PREFERENCE APPROACH TO THE ELICITATION OF POLITICAL ATTITUDES: EXPERIMENTAL EVIDENCE ON ANTI-AMERICANISM IN PAKISTAN , 2014 .

[41]  Jacob N. Shapiro,et al.  Poverty and Support for Militant Politics: Evidence from Pakistan , 2013 .

[42]  Adam Glynn.  What Can We Learn with Statistical Truth Serum?Design and Analysis of the List Experiment , 2013 .

[43]  Kosuke Imai,et al.  Statistical Analysis of List Experiments , 2012, Political Analysis.

[44]  Ezequiel Gonzalez-Ocantos,et al.  Vote Buying and Social Desirability Bias: Experimental Evidence from Nicaragua , 2012 .

[45]  M. Earleywine,et al.  Effect of response format on endorsement of eating disordered attitudes and behaviors. , 2007, The International journal of eating disorders.

[46]  Matthew N. Beckmann,et al.  What Leads to Voting Overreports? Contrasts of Overreporters to Validated Voters and Admitted Nonvoters in the American National Election Studies , 2001 .

[47]  Duane T. Wegener,et al.  Attitude change: Multiple roles for persuasion variables. , 1998 .

[48]  R. Cialdini.  Influence: The Psychology of Persuasion , 1993 .