Online psychophysics: reaction time effects in cognitive experiments

Using the Internet to acquire behavioral data is currently on the rise. However, very basic questions regarding the feasibility of online psychophysics are still open. Here, we aimed to replicate five well-known paradigms in experimental psychology (Stroop, Flanker, visual search, masked priming, attentional blink) in three settings (classical “lab”, “web-in-lab”, “web”) to account for possible changes in technology and environment. Lab and web-in-lab data were both acquired in an in-lab setting with lab using “Gold Standard” methods, while web-in-lab used web technology. This allowed for a direct comparison of potential differences in acquisition software. To account for additional environmental differences, the web technology experiments were published online to participate from home (setting web), thereby keeping the software and experimental design identical and only changing the environmental setting. Our main results are: First, we found an expected fixed additive timing offset when using web technology (M = 37 ms, SD = 8.14) and recording online (M = 87 ms, SD = 16.04) in comparison to lab data. Second, all task-specific effects were reproduced except for the priming paradigm, which couldn’t be replicated in any setting. Third, there were no differences in error rates, which are independent of the timing offset. This finding further supports the assumption of data equality over all settings. Fourth, we found that browser type might be influencing absolute reaction times. Together, these results contribute to the slowly but steadily growing literature that online psychophysics is a suitable complement – or even substitute – to lab data acquisition.

[1]  Colin M. Macleod.  Half a century of research on the Stroop effect: an integrative review. , 1991, Psychological bulletin.

[2]  Ian Neath,et al.  Response time accuracy in Apple Macintosh computers , 2011, Behavior research methods.

[3]  Martin Eimer,et al.  Links between conscious awareness and response inhibition: Evidence from masked priming , 2002, Psychonomic bulletin & review.

[4]  M. Birnbaum.  Introduction to psychological experiments on the internet , 2000 .

[5]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[6]  Michelle R. Greene,et al.  Visual search in scenes involves selective and nonselective pathways , 2011, Trends in Cognitive Sciences.

[7]  Stian Reimers,et al.  The BBC Internet Study: General Methodology , 2007, Archives of sexual behavior.

[8]  Jennifer Wiley,et al.  What Are the Odds? A Practical Guide to Computing and Reporting Bayes Factors , 2014, J. Probl. Solving.

[9]  W C Schmidt.  Presentation accuracy of Web animation methods , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[10]  Mark W. Bondi,et al.  Stroop Color‐Word Test , 2010 .

[11]  K. Forster,et al.  REPETITION PRIMING AND FREQUENCY ATTENUATION IN LEXICAL ACCESS , 1984 .

[12]  Neil Stewart,et al.  Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities , 2007, Behavior research methods.

[13]  A F Sanders,et al.  The Eriksen flanker effect revisited. , 2002, Acta psychologica.

[14]  J D Ramsey,et al.  Isodecrement curves for task performance in hot environments. , 1978, Applied ergonomics.

[15]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[16]  K L Shapiro,et al.  Temporary suppression of visual processing in an RSVP task: an attentional blink? . , 1992, Journal of experimental psychology. Human perception and performance.

[17]  Todd M. Gureckis,et al.  CUNY Academic , 2016 .

[18]  Diniz Lopes,et al.  ScriptingRT: A Software Library for Collecting Response Latencies in Online Studies of Cognition , 2013, PloS one.

[19]  R. Schvaneveldt,et al.  Facilitation in recognizing pairs of words: evidence of a dependence between retrieval operations. , 1971, Journal of experimental psychology.

[20]  Edward G. Sargis,et al.  The internet as psychological laboratory. , 2006, Annual review of psychology.

[21]  Tobias Elze,et al.  Temporal Properties of Liquid Crystal Displays: Implications for Vision Science Experiments , 2012, PloS one.

[22]  Benjamin A. Motz,et al.  Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task , 2015, Behavior Research Methods.

[23]  Neil Stewart,et al.  Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments , 2014, Behavior research methods.

[24]  Benjamin E Hilbig,et al.  Reaction time effects in lab- versus Web-based research: Experimental evidence , 2016, Behavior research methods.

[25]  M. Birnbaum.  Human research and data collection via the internet. , 2004, Annual review of psychology.

[26]  A. Joinson.  Self‐disclosure in computer‐mediated communication: The role of self‐awareness and visual anonymity , 2001 .

[27]  S. Gosling,et al.  Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. , 2004, The American psychologist.

[28]  R. Whelan.  Effective Analysis of Reaction Time Data , 2008 .

[29]  Erwin Haasnoot,et al.  QRTEngine: An easy solution for running online reaction time experiments using Qualtrics , 2014, Behavior research methods.

[30]  Ulf-Dietrich Reips.  Chapter 4 – The Web Experiment Method: Advantages, Disadvantages, and Solutions , 2000 .

[31]  Herbert H. Bell,et al.  On the relation between visual surround and motion aftereffect velocity , 1976 .

[32]  J. Windes,et al.  Reaction time for numerical coding and naming of numerals. , 1968, Journal of experimental psychology.

[33]  J. Darkes,et al.  Journal of the International Society of Sports Nutrition Open Access a League of Their Own: Demographics, Motivations and Patterns of Use of 1,955 Male Adult Non-medical Anabolic Steroid Users in the United States , 2007 .

[34]  Ben R. Newell,et al.  The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers , 2015, Judgment and Decision Making.

[35]  C. Eriksen,et al.  Effects of noise letters upon the identification of a target letter in a nonsearch task , 1974 .

[36]  K. Shapiro,et al.  The attentional blink , 1997, Trends in Cognitive Sciences.

[37]  K. Nakayama,et al.  Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments , 2012, Psychonomic Bulletin & Review.

[38]  Richard R Plant,et al.  Millisecond precision psychological research in a world of commodity computers: New hardware, new problems? , 2009, Behavior research methods.