QRTEngine: An easy solution for running online reaction time experiments using Qualtrics

Performing online behavioral research is gaining increased popularity among researchers in psychological and cognitive science. However, the currently available methods for conducting online reaction time experiments are often complicated and typically require advanced technical skills. In this article, we introduce the Qualtrics Reaction Time Engine (QRTEngine), an open-source JavaScript engine that can be embedded in the online survey development environment Qualtrics. The QRTEngine can be used to easily develop browser-based online reaction time experiments with accurate timing within current browser capabilities, and it requires only minimal programming skills. After introducing the QRTEngine, we briefly discuss how to create and distribute a Stroop task. Next, we describe a study in which we investigated the timing accuracy of the engine under different processor loads using external chronometry. Finally, we show that the QRTEngine can be used to reproduce classic behavioral effects in three reaction time paradigms: a Stroop task, an attentional blink task, and a masked-priming task. These findings demonstrate that QRTEngine can be used as a tool for conducting online behavioral research even when this requires accurate stimulus presentation times.

[1]  Diniz Lopes,et al.  ScriptingRT: A Software Library for Collecting Response Latencies in Online Studies of Cognition , 2013, PloS one.

[2]  Tara S. Behrend,et al.  The viability of crowdsourcing for survey research , 2011, Behavior research methods.

[3]  J. Ridley.  Studies of Interference in Serial Verbal Reactions , 2001 .

[4]  André Locher,et al.  Tatool: A Java-based open-source programming framework for psychological studies , 2013, Behavior research methods.

[5]  Martin Eimer,et al.  Links between conscious awareness and response inhibition: Evidence from masked priming , 2002, Psychonomic bulletin & review.

[6]  Colin M. Macleod.  Half a century of research on the Stroop effect: an integrative review. , 1991, Psychological bulletin.

[7]  Rolf Ulrich,et al.  Time resolution of clocks: Effects on reaction time measurement—Good news for bad clocks , 1989 .

[8]  Todd M. Gureckis,et al.  CUNY Academic , 2016 .

[9]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[10]  Travis Simcox,et al.  Collecting response times using Amazon Mechanical Turk and Adobe Flash , 2013, Behavior Research Methods.

[11]  Brian A. Nosek,et al.  Recommendations for Increasing Replicability in Psychology † , 2013 .

[12]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[13]  Jan Theeuwes,et al.  OpenSesame: An open-source, graphical experiment builder for the social sciences , 2011, Behavior Research Methods.

[14]  Martin Corley,et al.  Timing accuracy of Web experiments: A case study using the WebExp software package , 2009, Behavior research methods.

[15]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[16]  K L Shapiro,et al.  Temporary suppression of visual processing in an RSVP task: an attentional blink? . , 1992, Journal of experimental psychology. Human perception and performance.

[17]  K. Nakayama,et al.  Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments , 2012, Psychonomic Bulletin & Review.

[18]  Joshua de Leeuw,et al.  jsPsych: A JavaScript library for creating behavioral experiments in a Web browser , 2014, Behavior Research Methods.