Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments

Web-based research is becoming ubiquitous in the behavioral sciences, facilitated by convenient, readily available participant pools and relatively straightforward ways of running experiments: most recently, through the development of the HTML5 standard. Although in most studies participants give untimed responses, there is a growing interest in being able to record response times online. Existing data on the accuracy and cross-machine variability of online timing measures are limited, and generally they have compared behavioral data gathered on the Web with similar data gathered in the lab. For this article, we took a more direct approach, examining two ways of running experiments online—Adobe Flash and HTML5 with CSS3 and JavaScript—across 19 different computer systems. We used specialist hardware to measure stimulus display durations and to generate precise response times to visual stimuli in order to assess measurement accuracy, examining effects of duration, browser, and system-to-system variability (such as across different Windows versions), as well as effects of processing power and graphics capability. We found that (a) Flash and JavaScript’s presentation and response time measurement accuracy are similar; (b) within-system variability is generally small, even in low-powered machines under high load; (c) the variability of measured response times across systems is somewhat larger; and (d) browser type and system hardware appear to have relatively small effects on measured response times. Modeling of the effects of this technical variability suggests that for most within- and between-subjects experiments, Flash and JavaScript can both be used to accurately detect differences in response times across conditions. Concerns are, however, noted about using some correlational or longitudinal designs online.

[1]  Matthias Schwannauer,et al.  Psychological Processes Mediate the Impact of Familial Risk, Social Circumstances and Life Events on Mental Health , 2013, PloS one.

[2]  Philip T. Quinlan,et al.  Could millisecond timing errors in commonly used equipment be a cause of replication failure in some neuroscience studies? , 2013, Cognitive, Affective, & Behavioral Neuroscience.

[3]  Nick Hammond,et al.  Self-validating presentation and response timing in cognitive paradigms: How and why? , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[4]  Tom Whitehouse,et al.  How choice of mouse may affect response timing in psychological studies , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[5]  Diniz Lopes,et al.  ScriptingRT: A Software Library for Collecting Response Latencies in Online Studies of Cognition , 2013, PloS one.

[6]  Stian Reimers,et al.  The BBC Internet Study: General Methodology , 2007, Archives of sexual behavior.

[7]  Travis Simcox,et al.  Collecting response times using Amazon Mechanical Turk and Adobe Flash , 2013, Behavior Research Methods.

[8]  Ulf-Dietrich Reips,et al.  WEXTOR: A Web-based tool for generating and visualizing experimental designs and procedures , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[9]  S. Reimers,et al.  Task switching across the life span: effects of age on general and specific switch costs. , 2005, Developmental psychology.

[10]  Todd M. Gureckis,et al.  CUNY Academic , 2016 .

[11]  Martin Corley,et al.  Timing accuracy of Web experiments: A case study using the WebExp software package , 2009, Behavior research methods.

[12]  Carey K. Morewedge,et al.  Winners Love Winning and Losers Love Money , 2011, Psychological science.

[13]  James P. Evans.  The Origin , 2009, Genetics in Medicine.

[14]  Ulf-Dietrich Reips,et al.  A Brief History of Web Experimenting , 2000 .

[15]  M. Birnbaum.  Decision Making in the Lab and on the Web , 2000 .

[16]  Neil Stewart,et al.  Using Adobe Flash Lite on mobile phones for psychological research: Reaction time measurement reliability and interdevice variability , 2008, Behavior research methods.

[17]  Neil Stewart,et al.  Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities , 2007, Behavior research methods.

[18]  S. Moran,et al.  On the Origin , 2012 .

[19]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[20]  Ian Neath,et al.  Response time accuracy in Apple Macintosh computers , 2011, Behavior research methods.

[21]  Stian Reimers.  A paycheck half-empty or half-full? Framing, fairness and progressive taxation , 2009 .

[22]  Neil Stewart,et al.  On the Origin of Utility, Weighting, and Discounting Functions: How They Get Their Shapes and How to Change Their Shapes , 2015, Manag. Sci..

[23]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[24]  Andrew Brand,et al.  Assessing the Effects of Technical Variance on the Statistical Outcomes of Web Experiments Measuring Response Times , 2012 .

[25]  Markus F Damian,et al.  Does variability in human performance outweigh imprecision in response devices such as computer keyboards? , 2010, Behavior research methods.