Crowdsourcing research: Data collection with Amazon’s Mechanical Turk

ABSTRACT Researchers in a variety of disciplines use Amazon’s crowdsourcing platform called Mechanical Turk as a way to collect data from a respondent pool that is much more diverse than a typical student sample. The platform also provides cost efficiencies over other online panel services and data can be collected very quickly. However, some researchers have been slower to try the platform, perhaps because of a lack of awareness of its functions or concerns with validity. This article provides an overview of Mechanical Turk as an academic research platform and a critical examination of its strengths and weaknesses for research. Guidelines for collecting data that address issues of validity, reliability, and ethics are presented.

[1]  Florian Alexander Schmidt,et al.  The Good, The Bad and the Ugly: Why Crowdsourcing Needs Ethics , 2013, 2013 International Conference on Cloud and Green Computing.

[2]  K. Timpano,et al.  The importance of assessing clinical phenomena in Mechanical Turk research. , 2016, Psychological assessment.

[3]  Kim F. Nimon,et al.  A Primer for Conducting Survey Research using MTurk: Tips for the Field , 2016, Int. J. Adult Vocat. Educ. Technol..

[4]  U. Sailer,et al.  The affective profiles in the USA: happiness, depression, life satisfaction, and happiness-increasing strategies , 2013, PeerJ.

[5]  Daren C. Brabham.  MOVING THE CROWD AT THREADLESS , 2010 .

[6]  J. Siegel,et al.  The impact of overtly listing eligibility requirements on MTurk: An investigation involving organ donation, recruitment scripts, and feelings of elevation. , 2015, Social science & medicine.

[7]  K. Sheehan,et al.  An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk , 2017 .

[8]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[9]  K. D. Joshi,et al.  Is Crowdsourcing a Source of Worker Empowerment or Exploitation? Understanding Crowd Workers' Perceptions of Crowdsourcing Career , 2013, ICIS.

[10]  Daniel N. Jones,et al.  Introducing the Short Dark Triad (SD3) , 2014, Assessment.

[11]  Kate A. Ratliff,et al.  Using Nonnaive Participants Can Reduce Effect Sizes , 2015, Psychological science.

[12]  Leib Litman,et al.  The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk , 2014, Behavior Research Methods.

[13]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[14]  A. Barak,et al.  The benign online disinhibition effect: Could situational factors induce self-disclosure and prosocial behaviors? , 2015 .

[15]  Matthew Pittman,et al.  Amazon's Mechanical Turk for Academics: The HIT Handbook for Social Science Research , 2016 .

[16]  S. Levinson,et al.  WEIRD languages have misled us, too , 2010, Behavioral and Brain Sciences.

[17]  Jesse J. Chandler,et al.  Inside the Turk , 2014 .

[18]  Scott M. Smith,et al.  A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples , 2016 .

[19]  C. Chabris,et al.  Common (Mis)Beliefs about Memory: A Replication and Comparison of Telephone and Mechanical Turk Survey Methods , 2012, PloS one.

[20]  D. Tingley,et al.  “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents , 2015 .

[21]  Emily A. Cooper,et al.  Does the Sun revolve around the Earth? A comparison between the general public and online survey respondents in basic scientific knowledge , 2016, Public understanding of science.

[22]  K. Bretonnel Cohen,et al.  Last Words: Amazon Mechanical Turk: Gold Mine or Coal Mine? , 2011, CL.

[23]  Krista Casler,et al.  Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing , 2013, Comput. Hum. Behav..

[24]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[25]  D. Needham,et al.  Participant retention practices in longitudinal clinical research studies with high retention rates , 2017, BMC Medical Research Methodology.

[26]  R. Gardner,et al.  Using Amazon's Mechanical Turk website to measure accuracy of body size estimation and body dissatisfaction. , 2012, Body image.

[27]  Ben R. Newell,et al.  The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers , 2015, Judgment and Decision Making.

[28]  C. B. Colby.  The weirdest people in the world , 1973 .

[29]  Oded Netzer,et al.  MTurk Character Misrepresentation: Assessment and Solutions , 2017 .

[30]  P. Jonason,et al.  Walking the thin line between efficiency and accuracy: Validity and structural properties of the Dirty Dozen , 2013 .

[31]  Christopher J. Holden,et al.  Assessing the reliability of the M5-120 on Amazon's mechanical Turk , 2013, Comput. Hum. Behav..

[32]  Luis-Felipe Cabrera,et al.  AI Gets a Brain , 2006, ACM Queue.

[33]  M. Six Silberman,et al.  Turkopticon: interrupting worker invisibility in amazon mechanical turk , 2013, CHI.

[34]  Amar Cheema,et al.  Data collection in a flat world: the strengths and weaknesses of mechanical turk samples , 2013 .

[35]  A. Acquisti,et al.  Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research , 2016 .

[36]  Matthew Lease,et al.  Crowdsourcing for information retrieval: principles, methods, and applications , 2011, SIGIR.

[37]  Panagiotis G. Ipeirotis.  Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[38]  David J. Hauser,et al.  Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants , 2015, Behavior Research Methods.

[39]  Michael S. Bernstein,et al.  Mechanical Turk is Not Anonymous , 2013 .

[40]  Panagiotis G. Ipeirotis,et al.  The Global Opportunity in Online Outsourcing , 2015 .

[41]  J. Weisz,et al.  Using Mechanical Turk to Study Family Processes and Youth Mental Health: A Test of Feasibility , 2015, Journal of Child and Family Studies.

[42]  John B. Ford.  Amazon's Mechanical Turk: A Comment , 2017 .

[43]  Adam J. Berinsky,et al.  Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk , 2012, Political Analysis.

[44]  Samuel C. Lindsey,et al.  Practice-based considerations for using multi-stage survey design to reach special populations on Amazon’s Mechanical Turk , 2016 .

[45]  Heather Hessel,et al.  A Comparison of Three Online Recruitment Strategies for Engaging Parents. , 2015, Family relations.