SurveyMan

Surveys can be viewed as programs, complete with logic, control flow, and bugs. Word choice or the order in which questions are asked can unintentionally bias responses. Vague, confusing, or intrusive questions can cause respondents to abandon a survey. Surveys can also have runtime errors: inattentive respondents can taint results. This effect is especially problematic when deploying surveys in uncontrolled settings, such as on the web or via crowdsourcing platforms. Because the results of surveys drive business decisions and inform scientific conclusions, it is crucial to make sure they are correct. We present SurveyMan, a system for designing, deploying, and automatically debugging surveys. Survey authors write their surveys in a lightweight domain-specific language aimed at end users. SurveyMan statically analyzes the survey to provide feedback to survey authors before deployment. It then compiles the survey into JavaScript and deploys it either to the web or a crowdsourcing platform. SurveyMan's dynamic analyses automatically find survey bugs, and control for the quality of responses. We evaluate SurveyMan's algorithms analytically and empirically, demonstrating its effectiveness with case studies of social science surveys conducted via Amazon's Mechanical Turk.

[1]  S S Stevens,et al.  On the Theory of Scales of Measurement. , 1946, Science.

[2]  E. Glover.  DICTIONARY OF PSYCHOLOGY , 1959 .

[3]  Edsger W. Dijkstra,et al.  Guarded commands, nondeterminacy and formal derivation of programs , 1975, Commun. ACM.

[4]  Gilbert A. Churchill,et al.  Marketing Research: Methodological Foundations , 1976 .

[5]  D. Dillman,et al.  Mail and telephone surveys , 1978 .

[6]  B. Efron.  Bootstrap Methods: Another Look at the Jackknife , 1979 .

[7]  V. Rich.  Personal communication , 1989, Nature.

[8]  D. D. Vaus,et al.  Surveys in Social Research , 1991 .

[9]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[10]  Implementation , 2001 .

[11]  David J. Solomon,et al.  Conducting Web-based Surveys. , 2001 .

[12]  William B. Lober,et al.  SuML: A Survey Markup Language for Generalized Survey Encoding , 2002, AMIA.

[13]  Paul D. Umbach.  Web surveys: Best practices , 2004 .

[14]  Topsl : A domain-specific language for on-line surveys , 2004 .

[15]  Elizabeth Martin.  Survey Questionnaire Construction , 2005 .

[16]  Larry Wasserman,et al.  All of Nonparametric Statistics (Springer Texts in Statistics) , 2006 .

[17]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[18]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[19]  Ben Carterette,et al.  An Analysis of Assessor Behavior in Crowdsourced Preference Judgments , 2010 .

[20]  Lorrie Faith Cranor,et al.  Are your participants gaming the system?: screening mechanical turk workers , 2010, CHI.

[21]  Panagiotis G. Ipeirotis.  Demographics of Mechanical Turk , 2010 .

[22]  Andrew Raij,et al.  Exploring micro-incentive strategies for participant compensation in high-burden studies , 2011, UbiComp '11.

[23]  George MacKerron.  Implementation, implementation, implementation: old and new options for putting surveys and experiments online , 2011 .

[24]  A. Meade,et al.  Identifying careless responses in survey data. , 2012, Psychological methods.

[25]  Samuel C. Spencer.  A Case against the Skip Statement , 2012 .

[26]  Elizabeth M. Poposki,et al.  Detecting and Deterring Insufficient Effort Responding to Surveys , 2012 .

[27]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[28]  Adam J. Berinsky,et al.  Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk , 2012, Political Analysis.

[29]  Roger Tourangeau,et al.  The Science of Web Surveys , 2013 .

[30]  Alexander Scott McKay,et al.  IMPROVING DATA QUALITY WITH FOUR SHORT SENTENCES: HOW AN HONOR CODE CAN MAKE THE DIFFERENCE DURING DATA COLLECTION , 2014 .

[31]  J. P. Robinson-Cimpian,et al.  Inaccurate Estimation of Disparities Due to Mischievous Responders , 2014 .

[32]  Doreen Meier,et al.  Marketing Research Methodological Foundations , 2016 .

[33]  Andrew McGregor,et al.  AutoMan: a platform for integrating human-based and digital computation , 2012, OOPSLA '12.

[34]  Yvonne Jaeger,et al.  Designing Effective Web Surveys , 2016 .