Online Experiments using jsPsych, psiTurk, and Amazon Mechanical Turk

Online Experiments using jsPsych, psiTurk, and Amazon Mechanical Turk Joshua R. de Leeuw (jodeleeu@indiana.edu) Department of Psychological and Brain Sciences, Program in Cognitive Science, Indiana Univeristy, Bloomington, IN 47405 Anna Coenen (ac4066@nyu.edu) Doug Markant (doug.markant@nyu.edu) Jay B. Martin (jbmartin@nyu.edu) John V. McDonnell (jmcdon10@gmail.com) Alexander S. Rich (asr443@nyu.edu Todd M. Gureckis (todd.gureckis@nyu.edu) Department of Psychology, New York Univeristy New York, NY 10003 Part 1: What does the research say about online data collection? Keywords: Amazon Mechanical Turk; Online Experiments; jsPsych; psiTurk; Open Science This section of the tutorial will be a brief introduction to some of the issues surrounding AMT and online experiments, but the focus of the tutorial will be parts 2 and 3. Online experiments are appealing for a number of reasons: faster data collection, lower costs, access to a different subject pool, and improved anonymity of subjects and experimenters are some of the most commonly named. However, online ex- periments give up some of the control of a laboratory environ- ment, leading to concerns about the quality of the data. There are now several published results that compare AMT exper- iments to their laboratory counterparts (Paolacci, Chandler, & Ipeirotis, 2010; Buhrmester, Kwang, & Gosling, 2011; Zwann & Pechler, 2012; Crump, McDonnell, & Gureckis, 2013; Goodman, Cryder, & Cheema, 2013). In the tutorial, we will summarize these findings and their implications for running AMT based experiments. Objectives This half-day tutorial will cover how to build and deploy on- line experiments using jsPsych, psiTurk, and Amazon Me- chanical Turk (AMT). jsPsych is an open-source JavaScript library that facilitates building behavioral experiments in a web browser. psiTurk is an open-source Python platform that simplifies the process of running an experiment using AMT. Together, these two software packages reduce the complex- ity of setting up an online experiment on AMT, enabling researchers with little software programming experience to take advantage of online experiments. By the end of the tu- torial, participants will have gained hands-on experience in programming and deploying a basic behavioral experiment on AMT. Researchers in the Cognitive Science community have been using AMT, and online experiments in general, for sev- eral years, but the learning curve can be steep for researchers who are not familiar with web development. While some tools exist for certain kinds of simple experiments (e.g. ques- tionnaires), programming more complex experiments with dynamic elements requires knowledge of web-oriented pro- gramming. The tools covered in this tutorial simplify the pro- cess of programming online experiments, opening up the pos- sibility of conducting online experiments to more researchers in the cognitive science community. Workshops covering AMT (Mason & Suri, 2011) and psi- Turk (Coenen, Markant, Martin, & McDonnell, 2013) have been offered at previous Cognitive Science Society meetings. This tutorial goes one step further by covering jsPsych as well. Together, these tools cover the entire process of assem- bling and running an online experiment. Part 2: Assembling an experiment with jsPsych jsPsych (http://www.jspsych.org) is an open-source JavaScript library that simplifies the process of writing a web-browser-based experiment (de Leeuw, 2014). jsPsych contains a core library, which serves as the engine to run experiments, and a set of plugins, each of which defines a different kind of trial that a subject in an experiment might do. For example, there are plugins for displaying instructions, showing stimuli and collecting responses via the keyboard, and displaying a consent form. Assembling an experiment with jsPsych involves putting together the differ- ent plugins that are needed and specifying the parameters of those plugins (such as what stimuli to show and how long to show them). These plugins can be assembled to create many different behavioral tasks that are of interest to cognitive scientists. jsPsych can also be extended by writing new plugins. The structure of a plugin is flexible enough to permit most kinds of computer-based tasks. Because plugins are individual, stand- alone components of the library, each plugin can be combined with any of the others. As researchers use and extend jsPsych Outline of the Tutorial Participants at the tutorial will be invited to work hands-on with the creation of a simple online experiment that demon- strates the principles behind jsPsych and psiTurk. The tutorial will be organized in four parts.