A Primer for Conducting Survey Research using MTurk: Tips for the Field

This paper presents best practices for conducting survey research using Amazon Mechanical Turk MTurk. Readers will learn the benefits, limitations, and trade-offs of using MTurk as compared to other recruitment services, including SurveyMonkey and Qualtrics. A synthesis of survey design guidelines along with a sample survey are presented to help researchers collect the best quality data. Techniques, including SPSS and R syntax, are provided that demonstrate how users can clean resulting data and identify valid responses for which workers could be paid.

[1]  Robert W. Covert,et al.  Designing and Constructing Instruments for Social Research and Evaluation , 2007 .

[2]  Don A Dillman,et al.  Design effects in the transition to web-based surveys. , 2007, American journal of preventive medicine.

[3]  Gary Hsieh,et al.  You Get Who You Pay for: The Impact of Incentives on Participation Bias , 2016, CSCW.

[4]  Donald Tomaskovic-Devey,et al.  Organizational Survey Nonresponse , 1994 .

[5]  Russel L. Thompson,et al.  A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys , 2000 .

[6]  M. Banaji,et al.  Psychological. , 2015, The journals of gerontology. Series B, Psychological sciences and social sciences.

[7]  S. Rogelberg,et al.  Introduction Understanding and Dealing With Organizational Survey Nonresponse , 2007 .

[8]  Roger Tourangeau,et al.  What They See Is What We Get , 2004 .

[9]  Mick P. Couper,et al.  Web Survey Design Paging versus Scrolling , 2006 .

[10]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[11]  Thomas G. Reio,et al.  The Employee Engagement Landscape and HRD , 2011 .

[12]  Meredith Ringel Morris,et al.  Collaborative search revisited , 2013, CSCW.

[13]  Katerine Osatuke,et al.  Demographic Question Placement: Effect on Item Response Rates and Means of a Veterans Health Administration Survey , 2012 .

[14]  William E. Knight,et al.  Profiling active and passive nonrespondents to an organizational survey. , 2003, The Journal of applied psychology.

[15]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[16]  R. Carnegie.  Psychological Research Online : Opportunities and Challenges , 2003 .

[17]  Lydia B. Chilton,et al.  The labor economics of paid crowdsourcing , 2010, EC '10.

[18]  Mario Callegaro,et al.  Online Panel Research: A Data Quality Perspective , 2014 .

[19]  J. Shaw,et al.  Are financial incentives related to performance? A meta-analytic review of empirical research. , 1998 .

[20]  Marcia J. Simmering,et al.  A Tale of Three Perspectives , 2009 .

[21]  J. R. Larson,et al.  Research strategies and tactics in I/O psychology , 1990 .

[22]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[23]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[24]  D. Rousseau,et al.  Embracing translational HRD research for evidence-based management: let’s talk about how to bridge the research-practice gap , 2015 .

[25]  Joe E. Wheaton,et al.  Online Data Collection: Strategies for Research. , 2004 .

[26]  Zheng Yan,et al.  Factors affecting response rates of the web survey: A systematic review , 2010, Comput. Hum. Behav..

[27]  Brent Simpson,et al.  Emotional reactions to losing explain gender differences in entering a risky lottery , 2010, Judgment and Decision Making.

[28]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[29]  Olena Kaminska,et al.  Recruiting Probability Samples for a Multi-Mode Research Panel with Internet and Mail Components , 2010 .

[30]  Christine Nadel,et al.  Case Study Research Design And Methods , 2016 .

[31]  Thomas G. Reio,et al.  The Threat of Common Method Variance Bias to Theory Building , 2010 .

[32]  James H. Long,et al.  Online Instrument Delivery and Participant Recruitment Services: Emerging Opportunities for Behavioral Accounting Research , 2014 .

[33]  Krista Casler,et al.  Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing , 2013, Comput. Hum. Behav..

[34]  Mario Callegaro,et al.  Where Am I? A Meta-Analysis of Experiments on the Effects of Progress Indicators for Web Surveys , 2013 .

[35]  D. Lichtenstein,et al.  The Effect of Marketer-Suggested Serving Size on Consumer Responses: The Unintended Consequences of Consumer Attention to Calorie Information , 2012 .

[36]  Benjamin T. Hazen,et al.  Performance expectancy and use of enterprise architecture: training as an intervention , 2014, J. Enterp. Inf. Manag..

[37]  C. Spitzmueller,et al.  UNDERSTANDING RESPONSE BEHAVIOR TO AN ONLINE SPECIAL TOPICS ORGANIZATIONAL SATISFACTION SURVEY , 2006 .

[38]  Scott M. Smith,et al.  A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples , 2016 .

[39]  B. Thompson.  Foundations of behavioral statistics : an insight-based approach , 2006 .

[40]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[41]  Roy Y. J. Chua,et al.  The Costs of Ambient Cultural Disharmony: Indirect Intercultural Conflicts in Social Environment Undermine Creativity , 2013 .

[42]  I. Borg,et al.  Attitudes of demographic item non‐respondents in employee surveys , 2008 .

[43]  Katharina Reinecke,et al.  Crowdsourcing performance evaluations of user interfaces , 2013, CHI.

[44]  M. Prensky.  Digital Natives, Digital Immigrants Part 1 , 2001 .

[45]  David J. Ketchen,et al.  Addressing Common Method Variance: Guidelines for Survey Research on Information Technology, Operations, and Supply Chain Management , 2011, IEEE Transactions on Engineering Management.

[46]  Cihan Cobanoglu,et al.  The Effect of Incentives in Web Surveys: Application and Ethical Considerations , 2003 .

[47]  Andrea Hershatter,et al.  Millennials and the World of Work: An Organization and Management Perspective , 2010 .

[48]  Arjen van Witteloostuijn,et al.  From the Editors: Common method variance in international business research , 2010 .

[49]  Tara S. Behrend,et al.  The viability of crowdsourcing for survey research , 2011, Behavior research methods.

[50]  Philip G. Handwerk,et al.  On-Line vs. Paper-and-Pencil Surveying of Students: A Case Study. AIR 2000 Annual Forum Paper. , 2000 .

[51]  Adam J. Berinsky,et al.  Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk , 2012, Political Analysis.

[52]  Alan D. Mead,et al.  Inattentive Responding in MTurk and Other Online Samples , 2015, Industrial and Organizational Psychology.

[53]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[54]  C. Pannucci,et al.  Survey Says? A Primer on Web-Based Survey Design and Distribution , 2011, Plastic and reconstructive surgery.

[55]  Steven V. Rouse,et al.  A reliability analysis of Mechanical Turk data , 2015, Comput. Hum. Behav..

[56]  Kevin Crowston,et al.  Amazon Mechanical Turk: A Research Tool for Organizations and Information Systems Scholars , 2012, Shaping the Future of ICT Research.

[57]  Jesse Chandler,et al.  Using Mechanical Turk to Study Clinical Populations , 2013 .

[58]  Fernando R. Jiménez,et al.  Too Popular to Ignore: The Influence of Online Reviews on Purchase Intentions of Search and Experience Products , 2013 .

[59]  Christopher J. Holden,et al.  Assessing the reliability of the M5-120 on Amazon's mechanical Turk , 2013, Comput. Hum. Behav..

[60]  Richard N. Landers,et al.  An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other Convenience Samples , 2015, Industrial and Organizational Psychology.

[61]  Daniel A. Newman,et al.  Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English , 2015 .