The Role of MTurk in Education Research: Advantages, Issues, and Future Directions

The advent of online platforms such as Amazon’s Mechanical Turk (MTurk) has expanded considerably researchers’ options for collecting research data. Many researchers, however, express understandable skepticism of the viability of using platforms such as MTurk. In this article, we provide a background on the use of MTurk as a mechanism for collecting research data. We then review what is currently known about the advantages and issues associated with using MTurk and highlight important areas for future research. We conclude by discussing implications of the use of crowdsourcing platforms such as MTurk for education research.

[1]  Gianluca Demartini,et al.  It's getting crowded!: how to use crowdsourcing effectively for web science research , 2016, WebSci.

[2]  David J. Hauser,et al.  Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants , 2015, Behavior Research Methods.

[3]  Krista Casler,et al.  Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing , 2013, Comput. Hum. Behav..

[4]  Shi Xu,et al.  A Convenient Solution: Using MTurk To Sample From Hard-To-Reach Populations , 2015, Industrial and Organizational Psychology.

[5]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[6]  R. Peterson,et al.  Convenience samples of college students and research reproducibility , 2014 .

[7]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[8]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[9]  Steven V. Rouse,et al.  A reliability analysis of Mechanical Turk data , 2015, Comput. Hum. Behav..

[10]  A. Acquisti,et al.  Data Quality in Online Research : Challenges and Solutions , 2015 .

[11]  Christopher Williams.  Crowdsourcing Research: A Methodology for Investigating State Crime , 2013 .

[12]  Matthew Lease,et al.  Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms , 2015 .

[13]  Jesse Chandler,et al.  Using Mechanical Turk to Study Clinical Populations , 2013 .

[14]  Qinghua Zhu,et al.  Evaluation on crowdsourcing research: Current status and future direction , 2012, Information Systems Frontiers.

[15]  L. Williams,et al.  Crowdsourcing participants for psychological research in Australia: A test of Microworkers , 2017 .

[16]  Miguel Palacios,et al.  Crowdsourcing and organizational forms: Emerging trends and research implications , 2016 .

[17]  Gordon B. Schmidt,et al.  Fifty Days an MTurk Worker: The Social and Motivational Context for Amazon Mechanical Turk Workers , 2015, Industrial and Organizational Psychology.

[18]  A. Acquisti,et al.  Reputation as a sufficient condition for data quality on Amazon Mechanical Turk , 2013, Behavior Research Methods.

[19]  C. Bartneck,et al.  Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment , 2015, PloS one.

[20]  Robert Kosara,et al.  Do Mechanical Turks dream of square pie charts? , 2010, BELIV '10.

[21]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[22]  Daniel Burton Shank,et al.  Using Crowdsourcing Websites for Sociological Research: The Case of Amazon Mechanical Turk , 2016 .

[23]  Tara S. Behrend,et al.  The viability of crowdsourcing for survey research , 2011, Behavior research methods.

[24]  Adam J. Berinsky,et al.  Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk , 2012, Political Analysis.

[25]  Christopher J. Holden,et al.  Assessing the reliability of the M5-120 on Amazon's mechanical Turk , 2013, Comput. Hum. Behav..

[26]  Richard N. Landers,et al.  An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other Convenience Samples , 2015, Industrial and Organizational Psychology.

[27]  Amar Cheema,et al.  Data collection in a flat world: the strengths and weaknesses of mechanical turk samples , 2013 .