Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup

As the use of Amazon’s Mechanical Turk (MTurk) has increased among social science researchers, so, too, has research into the merits and drawbacks of the platform. However, while many endeavors have sought to address issues such as generalizability, the attentiveness of workers, and the quality of the associated data, there has been relatively less effort concentrated on integrating the various strategies that can be used to generate high-quality data using MTurk samples. Accordingly, the purpose of this research is twofold. First, existing studies are integrated into a set of strategies/best practices that can be used to maximize MTurk data quality. Second, focusing on task setup, selected platform-level strategies that have received relatively less attention in previous research are empirically tested to further enhance the contribution of the proposed best practices for MTurk usage.

[1]  S. Pijl,et al.  Loneliness among students with special educational needs in mainstream seventh grade. , 2012, Research in developmental disabilities.

[2]  Jeongdoo Park,et al.  Common method bias in hospitality research: A critical review of literature and an empirical study , 2016 .

[3]  Sarah C. Kucker,et al.  An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results , 2019, Social Psychological and Personality Science.

[4]  Adam Seth Levine,et al.  Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection , 2017 .

[5]  Scott M. Smith,et al.  A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples , 2016 .

[6]  Ainsworth Bailey,et al.  Consumer Awareness and Use of Product Review Websites , 2005 .

[7]  Kawon Kim,et al.  The Customer Isn’t Always Right: The Implications of Illegitimate Complaints , 2020 .

[8]  David J. Hauser,et al.  Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants , 2015, Behavior Research Methods.

[9]  The shape of and solutions to the MTurk quality crisis , 2020, Political Science Research and Methods.

[10]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[11]  Thomas J. Leeper,et al.  The Generalizability of Survey Experiments* , 2015, Journal of Experimental Political Science.

[12]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[13]  Scott Clifford,et al.  Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments , 2017, Comput. Hum. Behav..

[14]  Joseph Goodman,et al.  Crowdsourcing Consumer Research , 2017 .

[15]  E. King,et al.  Back to the Future , 2017, JACC. Cardiovascular interventions.

[16]  Aaron D. Shaw,et al.  Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India , 2012, CHI.

[17]  Haotian Zhou,et al.  of Personality and Social Psychology The Pitfall of Experimenting on the Web : How Unattended Selective Attrition Leads to Surprising ( Yet False ) Research Conclusions , 2016 .

[18]  Rodolfo Vázquez-Casielles,et al.  Satisfaction with service recovery: Perceived justice and emotional responses , 2009 .

[19]  Felix D. Schönbrodt,et al.  At what sample size do correlations stabilize , 2013 .

[20]  A. Mattila,et al.  Healthy Taste of High Status: Signaling Status at Restaurants , 2020, Cornell Hospitality Quarterly.

[21]  Jesse J. Chandler,et al.  Inside the Turk , 2014 .

[22]  Jesse Chandler,et al.  Conducting Clinical Research Using Crowdsourced Convenience Samples. , 2016, Annual review of clinical psychology.

[23]  E. Viding,et al.  Social Reward Questionnaire (SRQ): development and validation , 2014, Front. Psychol..

[24]  Oded Netzer,et al.  MTurk Character Misrepresentation: Assessment and Solutions , 2017 .

[25]  David G. Rand,et al.  Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk , 2017, Journal of the Economic Science Association.

[26]  K. D. Joshi,et al.  Why Individuals Participate in Micro-task Crowdsourcing Work Environment: Revealing Crowdworkers' Perceptions , 2016, J. Assoc. Inf. Syst..

[27]  Roi Reichart,et al.  The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers , 2020, Clinical Psychological Science.

[28]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[29]  John B. Ford Amazon's Mechanical Turk: A Comment , 2017 .

[30]  K. Sheehan,et al.  An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk , 2017 .

[31]  Amar Cheema,et al.  Data collection in a flat world: the strengths and weaknesses of mechanical turk samples , 2013 .

[32]  M. Six Silberman,et al.  From critical design to critical infrastructure: lessons from turkopticon , 2014, INTR.

[33]  Michael D. Buhrmester,et al.  An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use , 2018, Perspectives on psychological science : a journal of the Association for Psychological Science.

[34]  Steven V. Rouse,et al.  A reliability analysis of Mechanical Turk data , 2015, Comput. Hum. Behav..

[35]  A. Acquisti,et al.  Reputation as a sufficient condition for data quality on Amazon Mechanical Turk , 2013, Behavior Research Methods.

[36]  Nelson A. Barber,et al.  Customer Service Evaluations of Employees With Disabilities: The Roles of Perceived Competence and Service Failure , 2020, Cornell Hospitality Quarterly.

[37]  Justin A. DeSimone,et al.  Caution! MTurk Workers Ahead—Fines Doubled , 2015, Industrial and Organizational Psychology.

[38]  Marcia J. Simmering,et al.  Data Quality from Crowdsourced Surveys: A Mixed Method Inquiry into Perceptions of Amazon's Mechanical Turk Masters , 2018 .

[39]  Lin Guo,et al.  The Role of Perceived Control in Customer Value Cocreation and Service Recovery Evaluation , 2016 .

[40]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[41]  Shi Xu,et al.  A Convenient Solution: Using MTurk To Sample From Hard-To-Reach Populations , 2015, Industrial and Organizational Psychology.

[42]  Adam J. Berinsky,et al.  Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk , 2012, Political Analysis.

[43]  Chris Callison-Burch,et al.  A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk , 2017, CHI.