Running Behavioral Operations Experiments Using Amazon's Mechanical Turk

Mechanical Turk (MTurk), an online labor market run by Amazon, provides a platform for conducting behavioral experiments; the site offers immediate and inexpensive access to a large global subject pool. In this paper, we review recent research about MTurk and test the validity of using MTurk for experiments in behavioral operations management. We used subjects from MTurk to replicate the experiments in Bolton and Katok (2008), Engelbrecht-Wiggans and Katok (2008), Loch and Wu (2008), and Bolton, Ockenfels and Thonemann (2012). Our results are similar to the originals, but we also document some important differences. MTurk appears to be an important and relevant tool for researchers in behavioral operations, but we caution researchers to restrict use of this subject pool to experiments involving short-lived stimuli and behavioral manipulations.

[1]  Francis de Véricourt,et al.  Sex, Risk and the Newsvendor , 2013 .

[2]  Ulrich Wilhelm Thonemann,et al.  Managers and Students as Newsvendors , 2012, Manag. Sci..

[3]  Gérard P. Cachon,et al.  Decision Bias in the Newsvendor Problem with a Known Demand Distribution: Experimental Evidence.: Experimental Evidence. , 2000 .

[4]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[5]  Elena Katok,et al.  Regret and Feedback Information in First-Price Sealed-Bid Auctions , 2008, Manag. Sci..

[6]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[7]  Scott Clifford,et al.  Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies , 2014, Journal of Experimental Political Science.

[8]  J. Baron,et al.  Outcome bias in decision evaluation. , 1988, Journal of personality and social psychology.

[9]  David G. Rand,et al.  The promise of Mechanical Turk: how online labor markets can help theorists run behavioral experiments. , 2012, Journal of theoretical biology.

[10]  Yaozhong Wu,et al.  Social Preferences and Supply Chain Performance: An Experimental Study , 2008, Manag. Sci..

[11]  Sameer Hasija,et al.  Newsvendor pull-to-center reconsidered , 2014, Decis. Support Syst..

[12]  John A. Aloysius,et al.  Exploring Framing Effects in Inventory Control Decisions: Violations of Procedure Invariance , 2016 .

[13]  A. Tversky,et al.  Extensional versus intuitive reasoning: the conjunction fallacy in probability judgment , 1983 .

[14]  Krista Casler,et al.  Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing , 2013, Comput. Hum. Behav..

[15]  Gary E. Bolton,et al.  Learning-by-Doing in the Newsvendor Problem: A Laboratory Investigation of the Role of Experience and Feedback , 2008, Manuf. Serv. Oper. Manag..

[16]  David J. Hauser,et al.  Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants , 2015, Behavior Research Methods.

[17]  Brent Simpson,et al.  Emotional reactions to losing explain gender differences in entering a risky lottery , 2010, Judgment and Decision Making.

[18]  R. O. Chao,et al.  Tolerance for Failure and Incentives for Collaborative Innovation , 2013 .

[19]  A. Tversky,et al.  The framing of decisions and the psychology of choice. , 1981, Science.

[20]  Jesse J. Chandler,et al.  Inside the Turk , 2014 .

[21]  Lydia B. Chilton,et al.  The labor economics of paid crowdsourcing , 2010, EC '10.

[22]  Todd M. Gureckis,et al.  CUNY Academic , 2016 .

[23]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[24]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[25]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[26]  A. Acquisti,et al.  Reputation as a sufficient condition for data quality on Amazon Mechanical Turk , 2013, Behavior Research Methods.

[27]  John J. Horton Online Labor Markets , 2010 .

[28]  Tara S. Behrend,et al.  The viability of crowdsourcing for survey research , 2011, Behavior research methods.

[29]  George A. Akerlof Social Distance and Social Decisions , 1997 .

[30]  E. Fehr,et al.  Altruistic punishment in humans , 2002, Nature.