Conducting interactive experiments online

Online labor markets provide new opportunities for behavioral research, but conducting economic experiments online raises important methodological challenges. This particularly holds for interactive designs. In this paper, we provide a methodological discussion of the similarities and differences between interactive experiments conducted in the laboratory and online. To this end, we conduct a repeated public goods experiment with and without punishment using samples from the laboratory and the online platform Amazon Mechanical Turk. We chose to replicate this experiment because it is long and logistically complex. It therefore provides a good case study for discussing the methodological and practical challenges of online interactive experimentation. We find that basic behavioral patterns of cooperation and punishment in the laboratory are replicable online. The most important challenge of online interactive experiments is participant dropout. We discuss measures for reducing dropout and show that, for our case study, dropouts are exogenous to the experiment. We conclude that data quality for interactive experiments via the Internet is adequate and reliable, making online interactive experimentation a potentially valuable complement to laboratory studies.

[1]  David G. Rand,et al.  Inequality and visibility of wealth in experimental social networks , 2015, Nature.

[2]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[3]  David G. Rand,et al.  Social heuristics shape intuitive cooperation , 2014, Nature Communications.

[4]  Pablo Guillen,et al.  On “lab rats” , 2012 .

[5]  Simon Gächter,et al.  The limits of self-governance when cooperators get punished: Experimental evidence from urban and rural Russia , 2011 .

[6]  Reginald B. Adams,et al.  Investigating Variation in Replicability: A “Many Labs” Replication Project , 2014 .

[7]  David G. Rand,et al.  The promise of Mechanical Turk: how online labor markets can help theorists run behavioral experiments. , 2012, Journal of theoretical biology.

[8]  Stephen P. Jenkins,et al.  Easy Estimation Methods for Discrete-Time Duration Models , 1995 .

[9]  David G. Rand,et al.  Think global, act local: Preserving the global commons , 2016, Scientific Reports.

[10]  David G. Rand,et al.  Notes from a Day on the Forums: Recommendations for Maintaining a Good Reputation as an Amazon Mechanical Turk Requester , 2015 .

[11]  Matthew Haigh,et al.  Has the Standard Cognitive Reflection Test Become a Victim of Its Own Success? , 2016, Advances in cognitive psychology.

[12]  J. Abeler,et al.  Self-selection into laboratory experiments: pro-social motives versus monetary incentives , 2015 .

[13]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[14]  Duncan J. Watts,et al.  Cooperation and Contagion in Web-Based, Networked Public Goods Experiments , 2010, SECO.

[15]  Amar Cheema,et al.  Data collection in a flat world: the strengths and weaknesses of mechanical turk samples , 2013 .

[16]  Alessandro Acquisti,et al.  Beyond the Turk: An Empirical Comparison of Alternative Platforms for Crowdsourcing Online Behavioral Research , 2016 .

[17]  J. Carpenter,et al.  Do Social Preferences Increase Productivity? Field Experimental Evidence from Fishermen in Toyama Bay , 2005, SSRN Electronic Journal.

[18]  J. Freese,et al.  Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample , 2014 .

[19]  U. Fischbacher.  z-Tree: Zurich toolbox for ready-made economic experiments , 1999 .

[20]  Ulf-Dietrich Reips.  Chapter 4 – The Web Experiment Method: Advantages, Disadvantages, and Solutions , 2000 .

[21]  David G. Rand,et al.  Economic Games on the Internet: The Effect of $1 Stakes , 2011, PloS one.

[22]  Thomas J. Leeper,et al.  The Generalizability of Survey Experiments* , 2015, Journal of Experimental Political Science.

[23]  Tara S. Behrend,et al.  The viability of crowdsourcing for survey research , 2011, Behavior research methods.

[24]  Adam J. Berinsky,et al.  Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk , 2012, Political Analysis.

[25]  Andreas Nicklisch,et al.  hroot: Hamburg Registration and Organization Online Tool , 2014 .

[26]  Haotian Zhou,et al.  Journal of Personality and Social Psychology the Pitfall of Experimenting on the Web: How Unattended Selective Attrition Leads to Surprising (yet False) Research Conclusions , 2022 .

[27]  Edoardo Gallo,et al.  The effects of reputational and social knowledge on cooperation , 2015, Proceedings of the National Academy of Sciences.

[28]  Siddharth Suri,et al.  Cooperation and assortativity with dynamic partner updating , 2012, Proceedings of the National Academy of Sciences.

[29]  Christian Thöni,et al.  Trust, voluntary cooperation, and socio-economic background: survey and experimental evidence , 2004 .

[30]  Aldo Rustichini,et al.  Self-selection and variations in the laboratory measurement of other-regarding preferences across subject pools: evidence from one college student and two adult samples , 2013 .

[31]  A. Riedl,et al.  The economics of altruistic punishment and the maintenance of cooperation , 2008, Proceedings of the Royal Society B: Biological Sciences.

[32]  Jan Stoop,et al.  From the Lab to the Field: Cooperation among Fishermen , 2012, Journal of Political Economy.

[33]  Thomas Chesney,et al.  Virtual world experimentation: An exploratory study , 2009 .

[34]  Panagiotis G. Ipeirotis.  Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[35]  Stefan Pfattheicher,et al.  Understanding the Dark Side of Costly Punishment: The Impact of Individual Differences in Everyday Sadism and Existential Threat , 2015 .

[36]  Hawaii,et al.  Supporting Online Material Materials and Methods Figs. S1 to S6 Tables S1 and S2 Database S1 Antisocial Punishment across Societies , 2022 .

[37]  Rudolf Müller,et al.  Design and evaluation of an economic experiment via the Internet , 2001 .

[38]  Michèle Belot,et al.  Who should be called to the lab? A comprehensive comparison of students and non-students in classic experimental games , 2010 .

[39]  Nikos Nikiforakis,et al.  Is there selection bias in laboratory experiments? The case of social and risk preferences , 2013, SSRN Electronic Journal.

[40]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[41]  John A. List,et al.  Young, Selfish and Male: Field Evidence of Social Preferences , 2004 .

[42]  Panagiotis G. Ipeirotis,et al.  The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk , 2015, WWW.

[43]  E. Fehr,et al.  Altruistic punishment in humans , 2002, Nature.

[44]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[45]  Katrin Schmelz,et al.  Social Distance and Control Aversion : Evidence from the Internet and the Laboratory , 2015 .

[46]  Adam Seth Levine,et al.  Cross-Sample Comparisons and External Validity , 2014, Journal of Experimental Political Science.

[47]  Ben R. Newell,et al.  The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers , 2015, Judgment and Decision Making.

[48]  A. Acquisti,et al.  Reputation as a sufficient condition for data quality on Amazon Mechanical Turk , 2013, Behavior Research Methods.

[49]  David G. Rand,et al.  From good institutions to generous citizens: Top-down incentives to cooperate promote subsequent prosociality but not norm enforcement , 2017, Cognition.

[50]  Nicolas Jacquemet,et al.  Social preferences in the online laboratory: a randomized experiment , 2012 .

[51]  Kate A. Ratliff,et al.  Using Nonnaive Participants Can Reduce Effect Sizes , 2015, Psychological science.

[52]  Michal Krawczyk,et al.  What brings your subjects to the lab? A field experiment , 2011 .

[53]  Jesse J. Chandler,et al.  Inside the Turk , 2014 .

[54]  A. Acquisti,et al.  Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research , 2016 .

[55]  Ben Greiner,et al.  Subject pool recruitment procedures: organizing experiments with ORSEE , 2015, Journal of the Economic Science Association.

[56]  E. Fehr,et al.  Cooperation and Punishment in Public Goods Experiments , 1999, SSRN Electronic Journal.