psiTurk: An open-source framework for conducting replicable behavioral experiments online

Online data collection has begun to revolutionize the behavioral sciences. However, conducting carefully controlled behavioral experiments online introduces a number of new of technical and scientific challenges. The project described in this paper, psiTurk, is an open-source platform which helps researchers develop experiment designs which can be conducted over the Internet. The tool primarily interfaces with Amazon’s Mechanical Turk, a popular crowd-sourcing labor market. This paper describes the basic architecture of the system and introduces new users to the overall goals. psiTurk aims to reduce the technical hurdles for researchers developing online experiments while improving the transparency and collaborative nature of the behavioral sciences.

[1]  Todd M. Gureckis,et al.  Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research , 2013, PloS one.

[2]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[3]  Brian A. Nosek,et al.  An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[4]  Ulf-Dietrich Reips,et al.  WEXTOR: A Web-based tool for generating and visualizing experimental designs and procedures , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[5]  Ulf-Dietrich Reips Standards for Internet-based experimenting. , 2002, Experimental psychology.

[6]  Duncan J. Watts,et al.  Who says what to whom on twitter , 2011, WWW.

[7]  Rongjuan Chen,et al.  Perspective Matters: Sharing of Crisis Information in Social Media , 2013, 2013 46th Hawaii International Conference on System Sciences.

[8]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[9]  M. Birnbaum Human research and data collection via the internet. , 2004, Annual review of psychology.

[10]  Jeffrey T. Hancock,et al.  Experimental evidence of massive-scale emotional contagion through social networks , 2014, Proceedings of the National Academy of Sciences.

[11]  Ulf-Dietrich Reips Chapter 4 – The Web Experiment Method: Advantages, Disadvantages, and Solutions , 2000 .