The Number of Choice Tasks and Survey Satisficing in Conjoint Experiments

In recent years, political and social scientists have made increasing use of conjoint survey designs to study decision-making. Here, we study a consequential question which researchers confront when implementing conjoint designs: how many choice tasks can respondents perform before survey satisficing degrades response quality? To answer the question, we run a set of experiments where respondents are asked to complete as many as 30 conjoint tasks. Experiments conducted through Amazon's Mechanical Turk and Survey Sampling International demonstrate the surprising robustness of conjoint designs, as there are detectable but quite limited increases in survey satisficing as the number of tasks increases. Our evidence suggests that in similar study contexts researchers can assign dozens of tasks without substantial declines in response quality.

[1]  Lynn Vavreck,et al.  How Face-to-Face Interviews and Cognitive Skill Affect Item Non-Response: A Randomized Experiment Assigning Mode of Interview , 2016, Political Science Research and Methods.

[2]  J. Bachman,et al.  Effects of Questionnaire Length on Response Quality , 1981 .

[3]  J. Citrin,et al.  Public Attitudes Toward Immigration Policy Across the Legal/Illegal Divide: The Role of Categorical and Attribute-Based Decision-Making , 2016 .

[4]  Identifying Multidimensional Policy Preferences of Voters in Representative Democracies: A Conjoint Field Experiment in Japan , 2016 .

[5]  Dominik Hangartner,et al.  How economic, humanitarian, and religious concerns shape European attitudes toward asylum seekers , 2016, Science.

[6]  Daniel J. Hopkins,et al.  Beyond the breaking point? Survey satisficing in conjoint experiments , 2017, Political Science Research and Methods.

[7]  Marisa A. Abrajano,et al.  Using Experiments to Estimate Racially Polarized Voting , 2015 .

[8]  D. Yeager,et al.  Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples , 2011 .

[9]  Kenneth F. Scheve,et al.  Interests, Norms and Support for the Provision of Global Public Goods: The Case of Climate Cooperation , 2016 .

[10]  Vithala R. Rao,et al.  Conjoint Measurement- for Quantifying Judgmental Data , 1971 .

[11]  Damaraju Raghavarao,et al.  Choice-Based Conjoint Analysis: Models and Designs , 2010 .

[12]  G. Jasso,et al.  Distributive Justice and Earned Income , 1977 .

[13]  Daniel J. Hopkins,et al.  Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments , 2013 .

[14]  Ben R. Newell,et al.  The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers , 2015, Judgment and Decision Making.

[15]  J. Krosnick,et al.  Survey research. , 1999, Annual review of psychology.

[16]  Clayton Nall,et al.  Why Partisans Do Not Sort: The Constraints on Political Segregation , 2017, The Journal of Politics.

[17]  M. Galesic,et al.  Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey , 2009 .

[18]  Jens Hainmueller,et al.  Validating vignette and conjoint survey experiments against real-world behavior , 2015, Proceedings of the National Academy of Sciences.

[19]  Fabio Franchino,et al.  Voting in a Multi-dimensional Space: A Conjoint Analysis Employing Valence and Ideology Attributes of Candidates , 2014, Political Science Research and Methods.

[20]  Jens Hainmueller,et al.  The Hidden American Immigration Consensus: A Conjoint Analysis of Attitudes Toward Immigrants , 2012 .

[21]  J. Krosnick,et al.  National Surveys Via Rdd Telephone Interviewing Versus the Internet Comparing Sample Representativeness and Response Quality , 2009 .

[22]  The Number of Choice Tasks and Survey Satisficing in Conjoint Experiments , 2018, Political Analysis.

[23]  M. Sen,et al.  Analyzing Causal Mechanisms in Survey Experiments , 2018, Political Analysis.

[24]  Kosuke Imai,et al.  Causal Interaction in High-Dimension ∗ , 2015 .

[25]  Adam J. Berinsky,et al.  Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys , 2014 .

[26]  D. Kahneman A perspective on judgment and choice: mapping bounded rationality. , 2003, The American psychologist.