The promise of Mechanical Turk: how online labor markets can help theorists run behavioral experiments.

Combining evolutionary models with behavioral experiments can generate powerful insights into the evolution of human behavior. The emergence of online labor markets such as Amazon Mechanical Turk (AMT) allows theorists to conduct behavioral experiments very quickly and cheaply. The process occurs entirely over the computer, and the experience is quite similar to performing a set of computer simulations. Thus AMT opens the world of experimentation to evolutionary theorists. In this paper, I review previous work combining theory and experiments, and I introduce online labor markets as a tool for behavioral experimentation. I review numerous replication studies indicating that AMT data is reliable. I also present two new experiments on the reliability of self-reported demographics. In the first, I use IP address logging to verify AMT subjects' self-reported country of residence, and find that 97% of responses are accurate. In the second, I compare the consistency of a range of demographic variables reported by the same subjects across two different studies, and find between 81% and 98% agreement, depending on the variable. Finally, I discuss limitations of AMT and point out potential pitfalls. I hope this paper will encourage evolutionary modelers to enter the world of experimentation, and help to strengthen the bond between theoretical and empirical analyses of the evolution of human behavior.

[1]  David G. Rand,et al.  Direct reciprocity with costly punishment: generous tit-for-tat prevails. , 2009, Journal of theoretical biology.

[2]  Ernst Fehr,et al.  Are people conditionally cooperative , 2000 .

[3]  Terence C. Burnham,et al.  Engineering altruism: a theoretical and experimental investigation of anonymity and gift giving , 2003 .

[4]  M. Milinski,et al.  Cooperation through image scoring in humans. , 2000, Science.

[5]  Benedikt Herrmann,et al.  The Moral Costs of Nastiness , 2011 .

[6]  U. Fischbacher,et al.  Are People Conditionally Cooperative? Evidence from a Public Goods Experiment , 2001 .

[7]  BÓ Pedrodal,et al.  Cooperation under the Shadow of the Future : Experimental Evidence from Infinitely Repeated Games , 2005 .

[8]  Terence C Burnham,et al.  Engineering Human Cooperation , 2007, Human nature.

[9]  M. Milinski,et al.  Reputation helps solve the ‘tragedy of the commons’ , 2002, Nature.

[10]  Drew Fudenberg,et al.  The Folk Theorem in Repeated Games with Discounting or with Incomplete Information , 1986 .

[11]  Arthur Schram,et al.  Indirect Punishment and Generosity Toward Strangers , 2009, Science.

[12]  Drew Fudenberg,et al.  Who Cooperates in Repeated Games , 2011 .

[13]  M. Milinski,et al.  Human cooperation in the simultaneous and the alternating Prisoner's Dilemma: Pavlov versus Generous Tit-for-Tat. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[15]  Arne Traulsen,et al.  Social learning promotes institutions for governing the commons , 2010, Nature.

[16]  Bettina Rockenbach,et al.  The efficient interaction of indirect reciprocity and costly punishment , 2006, Nature.

[17]  R. Bshary,et al.  The interplay of cognition and cooperation , 2010, Philosophical Transactions of the Royal Society B: Biological Sciences.

[18]  J. Brooks.  Why most published research findings are false: Ioannidis JP, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece , 2008 .

[19]  David G. Rand,et al.  Slow to Anger and Fast to Forgive: Cooperation in an Uncertain World , 2010 .

[20]  Rachel Croson,et al.  Thinking like a game theorist: factors affecting the frequency of equilibrium play , 2000 .

[21]  Matthias Sutter,et al.  Choosing the Stick or the Carrot? Endogenous Institutional Choice in Social Dilemma Situations , 2006 .

[22]  Nikos Nikiforakis,et al.  Punishment and Counter-punishment in Public Good Games: Can We Still Govern Ourselves? , 2005 .

[23]  P. Ubel,et al.  Measuring Numeracy without a Math Test: Development of the Subjective Numeracy Scale , 2007, Medical decision making : an international journal of the Society for Medical Decision Making.

[24]  Bryan Hattingh,et al.  The competitive advantage , 2007 .

[25]  ปิยดา สมบัติวัฒนา.  Behavioral Game Theory: Experiments in Strategic Interaction , 2013 .

[26]  David G. Rand,et al.  Anti-social Punishment Can Prevent the Co-evolution of Punishment and Cooperation , 2010 .

[27]  Lourdes Araujo,et al.  Social Experiments in the Mesoscale: Humans Playing a Spatial Prisoner's Dilemma , 2010, PloS one.

[28]  C. Hauert,et al.  Volunteering as Red Queen Mechanism for Cooperation in Public Goods Games , 2002, Science.

[29]  David G. Rand,et al.  Who Cooperates in Repeated Games: The Role of Altruism, Inequity Aversion, and Demographics , 2011 .

[30]  James M. Walker,et al.  Rewards and Sanctions and the Provision of Public Goods in One-Shot Settings , 2004 .

[31]  D. Fudenberg,et al.  Evolution and Cooperation in Noisy Repeated Games , 1990 .

[32]  T. Yamagishi.  The provision of a sanctioning system as a public good , 1986 .

[33]  Dirk Semmann,et al.  Reputation is valuable within and outside one’s own social group , 2005, Behavioral Ecology and Sociobiology.

[34]  M. Nowak.  Five Rules for the Evolution of Cooperation , 2006, Science.

[35]  P. Bó.  Cooperation under the Shadow of the Future: Experimental Evidence from Infinitely Repeated Games , 2005 .

[36]  Yi Tao,et al.  Costly punishment does not always increase cooperation , 2009, Proceedings of the National Academy of Sciences.

[37]  M. Nowak,et al.  Evolution of indirect reciprocity , 2005, Nature.

[38]  Simon Gächter,et al.  Reciprocity, culture and human cooperation: previous insights and a new cross-cultural experiment , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[39]  Simon Gächter,et al.  The effects of (incentivized) belief elicitation in public goods experiments , 2010 .

[40]  M. Nowak,et al.  Tit for tat in heterogeneous populations , 1992, Nature.

[41]  David G. Rand,et al.  Winners don’t punish , 2008, Nature.

[42]  F. Schwartz.  The Competitive Advantage , 1996 .

[43]  M. Milinski,et al.  Volunteering leads to rock–paper–scissors dynamics in a public goods game , 2003, Nature.

[44]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[45]  U. Fischbacher,et al.  Are People Conditionally Cooperative? Evidence from a Public Goods Experiment , 2001 .

[46]  Matthias Sutter,et al.  Choosing the Carrot or the Stick? Endogenous Institutional Choice in Social Dilemma Situations , 2010 .

[47]  M. Milinski,et al.  Cooperation through indirect reciprocity: image scoring or standing strategy? , 2001, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[48]  Hawaii,et al.  Supporting Online Material Materials and Methods Figs. S1 to S6 Tables S1 and S2 Database S1 Antisocial Punishment across Societies , 2022 .

[49]  Roberto Ricciuti,et al.  Punishment and Counter-punishment in Public Goods Games: Can we still govern ourselves? , 2004 .

[50]  A. Melis,et al.  How is human cooperation different? , 2010, Philosophical Transactions of the Royal Society B: Biological Sciences.

[51]  Yoh Iwasa,et al.  The leading eight: social norms that can maintain cooperation by indirect reciprocity. , 2006, Journal of theoretical biology.

[52]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[53]  Benjamin C. Campbell,et al.  The 7R polymorphism in the dopamine receptor D4 gene (DRD4) is associated with financial risk taking in men , 2009 .

[54]  Abdolkarim Sadrieh,et al.  The pleasure of being nasty , 2009 .

[55]  William T. Harbaugh,et al.  The Carrot or the Stick: Rewards, Punishments and Cooperation , 2002 .

[56]  S. Zamir,et al.  Bargaining and Market Behavior in Jerusalem, Ljubljana, Pittsburgh, and Tokyo: An Experimental Study , 1991 .

[57]  Robert Shupp,et al.  The Effect of Rewards and Sanctions in Provision of Public Goods , 2006 .

[58]  David G. Rand,et al.  Decision-Making in Research Tasks with Sequential Testing , 2009, PloS one.

[59]  W. Hamilton,et al.  The Evolution of Cooperation , 1984 .

[60]  E. Fehr,et al.  Cooperation and Punishment in Public Goods Experiments , 1999, SSRN Electronic Journal.

[61]  Arne Traulsen,et al.  Human strategy updating in evolutionary games , 2010, Proceedings of the National Academy of Sciences.

[62]  Josef Hofbauer,et al.  Evolutionary Games and Population Dynamics , 1998 .

[63]  Arne Traulsen,et al.  Exploration dynamics in evolutionary games , 2009, Proceedings of the National Academy of Sciences.

[64]  Marco A Janssen,et al.  Evolution of cooperation and altruistic punishment when retaliation is possible , 2008 .

[65]  Alexander K. Koch,et al.  Belief elicitation in experiments: is there a hedging problem? , 2010 .

[66]  David G. Rand,et al.  Third Party Reward and Punishment: Group Size, Efficiency and Public Goods , 2010 .

[67]  Michihiro Kandori.  Social Norms and Community Enforcement , 1992 .

[68]  M. Nowak,et al.  A strategy of win-stay, lose-shift that outperforms tit-for-tat in the Prisoner's Dilemma game , 1993, Nature.

[69]  C. Apicella,et al.  Testosterone and financial risk preferences , 2008 .

[70]  David M. Kreps,et al.  Reputation and imperfect information , 1982 .

[71]  David G. Rand,et al.  Positive Interactions Promote Public Cooperation , 2009, Science.

[72]  A. Schram,et al.  Social status and group norms: Indirect reciprocity in a repeated helping experiment , 2006 .

[73]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[74]  D. Fessler,et al.  Nobody's watching? Subtle cues affect generosity in an anonymous economic game. , 2005 .

[75]  B. Rockenbach,et al.  The Competitive Advantage of Sanctioning Institutions , 2006, Science.

[76]  D. Prelec.  A Bayesian Truth Serum for Subjective Data , 2004, Science.

[77]  David Masclet,et al.  Punishment, counterpunishment and sanction enforcement in a social dilemma experiment , 2007 .

[78]  M. Nowak.  Evolutionary Dynamics: Exploring the Equations of Life , 2006 .

[79]  Guillaume Fréchette,et al.  The Evolution of Cooperation in Infinitely Repeated Games: Experimental Evidence , 2011 .

[80]  E. Hill.  Journal of Theoretical Biology , 1961, Nature.

[81]  Bettina Rockenbach,et al.  On the interaction of the stick and the carrot in social dilemmas. , 2012, Journal of theoretical biology.

[82]  Duncan J. Watts,et al.  Cooperation and Contagion in Web-Based, Networked Public Goods Experiments , 2010, SECO.

[83]  Louis Putterman,et al.  Can second-order punishment deter perverse punishment? , 2006 .

[84]  William T. Harbaugh,et al.  Children's altruism in public good and dictator experiments , 2000 .

[85]  Simon Gächter,et al.  The limits of self-governance when cooperators get punished: Experimental evidence from urban and rural Russia , 2011 .