TaskMaster: A Tool for Determining When Subjects Are on Task

In this Tutorial, we introduce a tool that allows researchers to track subjects’ on- and off-task activity on Qualtrics’ online survey platform. Our TaskMaster tool uses JavaScript to create several arrayed variables representing the frequency with which subjects enter and leave an active survey window and how long they remain within a given window. We provide code and instructions that will allow researchers to both implement the TaskMaster and analyze its output. We detail several potential applications, such as in studies of persistence and cheating, and studies that require sustained attention to experimental outcomes. The TaskMaster is designed to be accessible to researchers who are comfortable designing studies in Qualtrics, but who may have limited experience using programming languages such as JavaScript.

[1]  Joshua de Leeuw,et al.  jsPsych: A JavaScript library for creating behavioral experiments in a Web browser , 2014, Behavior Research Methods.

[2]  Scott Clifford,et al.  Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments , 2017, Comput. Hum. Behav..

[3]  Carsten Jensen,et al.  Self-reported cheating in web surveys on political knowledge , 2014 .

[4]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[5]  N. Taatgen,et al.  The problem state: a cognitive bottleneck in multitasking. , 2010, Journal of experimental psychology. Learning, memory, and cognition.

[6]  A. Acquisti,et al.  Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research , 2016 .

[7]  Scott Clifford,et al.  Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies , 2014, Journal of Experimental Political Science.

[8]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[9]  Robert C. Luskin,et al.  ''Don't Know'' Means ''Don't Know'': DK Responses and the Public's Level of Political Knowledge , 2011 .

[10]  Jesse Chandler,et al.  Common Concerns with MTurk as a Participant Pool , 2018, Handbook of Research Methods in Consumer Psychology.

[11]  P. Curran.  Methods for the detection of carelessly invalid responses in survey data , 2016 .

[12]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[13]  Birk Diedenhofen,et al.  PageFocus: Using paradata to detect and prevent cheating on online achievement tests , 2017, Behavior research methods.