Conducting perception research over the internet: a tutorial review

This article provides an overview of the recent literature on the use of internet-based testing to address important questions in perception research. Our goal is to provide a starting point for the perception researcher who is keen on assessing this tool for their own research goals. Internet-based testing has several advantages over in-lab research, including the ability to reach a relatively broad set of participants and to quickly and inexpensively collect large amounts of empirical data, via services such as Amazon’s Mechanical Turk or Prolific Academic. In many cases, the quality of online data appears to match that collected in lab research. Generally-speaking, online participants tend to be more representative of the population at large than those recruited for lab based research. There are, though, some important caveats, when it comes to collecting data online. It is obviously much more difficult to control the exact parameters of stimulus presentation (such as display characteristics) with online research. There are also some thorny ethical elements that need to be considered by experimenters. Strengths and weaknesses of the online approach, relative to others, are highlighted, and recommendations made for those researchers who might be thinking about conducting their own studies using this increasingly-popular approach to research in the psychological sciences.

[1]  H. Pashler,et al.  Editors’ Introduction to the Special Section on Replicability in Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[2]  M. Banaji,et al.  Psychological. , 2015, The journals of gerontology. Series B, Psychological sciences and social sciences.

[3]  Diniz Lopes,et al.  ScriptingRT: A Software Library for Collecting Response Latencies in Online Studies of Cognition , 2013, PloS one.

[4]  Carlos Velasco,et al.  Assessing the Role of Taste Intensity and Hedonics in Taste-Shape Correspondences. , 2016, Multisensory research.

[5]  Alessandro Acquisti,et al.  Beyond the Turk: An Empirical Comparison of Alternative Platforms for Crowdsourcing Online Behavioral Research , 2016 .

[6]  C. Spence,et al.  Exploring implicit and explicit crossmodal colour-flavour correspondences in product packaging , 2012 .

[7]  Jesse Chandler,et al.  Risks and Rewards of Crowdsourcing Marketplaces , 2014, Handbook of Human Computation.

[8]  C. Spence,et al.  Does the type of receptacle influence the crossmodal association between colour and flavour? A cross-cultural comparison , 2014, Flavour.

[9]  Jeffrey Witzel,et al.  Testing the viability of webDMDX for masked priming experiments , 2014 .

[10]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[11]  David J. Hauser,et al.  Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants , 2015, Behavior Research Methods.

[12]  I. Cross Music, Cognition, Culture, and Evolution , 2001, Annals of the New York Academy of Sciences.

[13]  K. Nakayama,et al.  Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments , 2012, Psychonomic Bulletin & Review.

[14]  M. J. Intons-Peterson,et al.  Imagery paradigms: how vulnerable are they to experimenters' expectations? , 1983, Journal of experimental psychology. Human perception and performance.

[15]  Travis Simcox,et al.  Collecting response times using Amazon Mechanical Turk and Adobe Flash , 2013, Behavior Research Methods.

[16]  Phillip Atiba Goff,et al.  Clearing the air: the effect of experimenter race on target's test performance and subjective experience. , 2005, The British journal of social psychology.

[17]  Michael S. Bernstein,et al.  Mechanical Turk is Not Anonymous , 2013 .

[18]  Joshua de Leeuw,et al.  jsPsych: A JavaScript library for creating behavioral experiments in a Web browser , 2014, Behavior Research Methods.

[19]  Amar Cheema,et al.  Data collection in a flat world: the strengths and weaknesses of mechanical turk samples , 2013 .

[20]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[21]  Reginald B. Adams,et al.  Investigating Variation in Replicability: A “Many Labs” Replication Project , 2014 .

[22]  D. Gilbert,et al.  A Wandering Mind Is an Unhappy Mind , 2010, Science.

[23]  C. Eriksen,et al.  The flankers task and response competition: A useful tool for investigating a variety of cognitive problems , 1995 .

[24]  C. Hendrick,et al.  Experimenter sex effects in behavioral research. , 1977 .

[25]  R. Shepard,et al.  Learning and memorization of classifications. , 1961 .

[26]  Leif D. Nelson,et al.  P-Curve: A Key to the File Drawer , 2013, Journal of experimental psychology. General.

[27]  Ian Watkins,et al.  Perception problems of the verbal scale: A reanalysis and application of a membership function approach. , 2015, Science & justice : journal of the Forensic Science Society.

[28]  Carlos Velasco,et al.  The context of colour-flavour associations in crisps packaging: A cross-cultural study comparing Chinese, Colombian, and British consumers , 2014 .

[29]  Eli Peli,et al.  Psychophysical contrast calibration , 2013, Vision Research.

[30]  T. Stafford,et al.  Tracing the Trajectory of Skill Learning With a Very Large Sample of Online Game Players , 2014, Psychological science.

[31]  Rossano Schifanella,et al.  Friendship prediction and homophily in social media , 2012, TWEB.

[32]  R. Nosofsky,et al.  Comparing modes of rule-based classification learning: A replication and extension of Shepard, Hovland, and Jenkins (1961) , 1994, Memory & cognition.

[33]  Vitaly Shmatikov,et al.  De-anonymizing Social Networks , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[34]  J. Ziegler,et al.  Smart Phone, Smart Science: How the Use of Smartphones Can Revolutionize Research in Cognitive Science , 2011, PloS one.

[35]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[36]  Massimiliano Di Luca,et al.  Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. , 2009, Journal of vision.

[37]  Todd M. Gureckis,et al.  CUNY Academic , 2016 .

[38]  C. Levitan,et al.  Red hot: the crossmodal effect of color intensity on perceived piquancy. , 2014, Multisensory research.

[39]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[40]  F. Vallée-Tourangeau,et al.  How to train your Bayesian: a problem-representation transfer rather than a format-representation shift explains training effects. , 2015, Quarterly journal of experimental psychology.

[41]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[42]  David G. Rand,et al.  Social heuristics shape intuitive cooperation , 2014, Nature Communications.

[43]  Mario Andrés Paredes-Valverde,et al.  A systematic review of tools, languages, and methodologies for mashup development , 2015, Softw. Pract. Exp..

[44]  D. Wechsler,et al.  Wechsler Adult Intelligence Scale—Fourth Edition (WAIS-IV) , 2010 .

[45]  H. Busher,et al.  Ethical issues in online research , 2015 .

[46]  Andrew Brand,et al.  Assessing the Effects of Technical Variance on the Statistical Outcomes of Web Experiments Measuring Response Times , 2012 .

[47]  M. Orne On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. , 1962 .

[48]  Feng Zhao,et al.  Gibraltar: Exposing Hardware Devices to Web Pages Using AJAX , 2012, WebApps.

[49]  Dorret I. Boomsma,et al.  Accounting for sequential trial effects in the flanker task: Conflict adaptation or associative priming? , 2006, Memory & cognition.

[50]  Panagiotis G. Ipeirotis Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[51]  Kimron Shapiro,et al.  Attentional blink , 2009, Scholarpedia.

[52]  Diego López-de-Ipiña,et al.  Measuring Software Timing Errors in the Presentation of Visual Stimuli in Cognitive Neuroscience Experiments , 2014, PloS one.

[53]  Neil Stewart,et al.  Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments , 2014, Behavior research methods.

[54]  Neil Stewart,et al.  Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities , 2007, Behavior research methods.

[55]  Rick A Adams,et al.  Crowdsourcing for Cognitive Science – The Utility of Smartphones , 2014, PloS one.

[56]  G. King,et al.  Ensuring the Data-Rich Future of the Social Sciences , 2011, Science.

[57]  Richard J. Brown,et al.  Brief body-scan meditation practice improves somatosensory perceptual decision making , 2012, Consciousness and Cognition.

[58]  Anton Nijholt Breaking Fresh Ground in Human–Media Interaction Research , 2014, Front. ICT.

[59]  Charles Spence,et al.  That sounds sweet: using cross-modal correspondences to communicate gustatory attributes , 2015 .

[60]  Khaled El Emam,et al.  Anonymizing Health Data: Case Studies and Methods to Get You Started , 2013 .

[61]  Peter Totterdell,et al.  Mind-wandering and negative mood: Does one thing really lead to another? , 2013, Consciousness and Cognition.

[62]  P. Kay,et al.  Resolving the question of color naming universals , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[63]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[64]  C. Spence,et al.  When the shape of the glass influences the flavour associated with a coloured beverage: Evidence from consumers in three countries , 2015 .

[65]  C. Spence,et al.  Hedonic mediation of the crossmodal correspondence between taste and shape , 2015 .

[66]  A Preliminary Study of Daily Sample Composition on Amazon Mechanical Turk , 2015 .

[67]  Barbara A. Spellman,et al.  Introduction to the Special Section , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[68]  S. Shimojo,et al.  Visual illusion induced by sound. , 2002, Brain research. Cognitive brain research.

[69]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[70]  M. Eimer,et al.  Effects of masked stimuli on motor activation: behavioral and electrophysiological evidence. , 1998, Journal of experimental psychology. Human perception and performance.

[71]  Sanne Boesveldt,et al.  Cross-Cultural Color-Odor Associations , 2014, PloS one.

[72]  Axel Cleeremans,et al.  Behavioral Priming: It's All in the Mind, but Whose Mind? , 2012, PloS one.

[73]  Ian Neath,et al.  Response time accuracy in Apple Macintosh computers , 2011, Behavior research methods.

[74]  C. Spence Audiovisual multisensory integration , 2007 .

[75]  John H. Krantz,et al.  Stimulus Delivery on the Web : What Can Be Presented when Calibration isn ’ t Possible , 2001 .

[76]  C. B. Colby The weirdest people in the world , 1973 .

[77]  T. Buchanan,et al.  Ethics Guidelines for Internet-mediated Research , 2013 .

[78]  P. Fitzgerald Gray colored glasses: is major depression partially a sensory perceptual disorder? , 2013, Journal of affective disorders.

[79]  Ben Bauer,et al.  A timely reminder about stimulus display times and other presentation parameters on CRTs and newer technologies. , 2015, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[80]  S. Levinson,et al.  WEIRD languages have misled us, too , 2010, Behavioral and Brain Sciences.

[81]  Jesse J. Chandler,et al.  Inside the Turk , 2014 .

[82]  Lorrie Faith Cranor,et al.  Are your participants gaming the system?: screening mechanical turk workers , 2010, CHI.

[83]  Carlos Velasco,et al.  Searching for flavor labels in food products: the influence of color-flavor congruence and association strength , 2015, Front. Psychol..

[84]  Jasper J. F. van den Bosch,et al.  Cross-cultural differences in crossmodal correspondences between basic tastes and visual features , 2014, Front. Psychol..

[85]  Lindsay T. Graham,et al.  A Review of Facebook Research in the Social Sciences , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[86]  P. Kay Basic Color Terms: Their Universality and Evolution , 1969 .

[87]  Vitaly Shmatikov,et al.  Robust De-anonymization of Large Sparse Datasets , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).

[88]  C Shawn Green,et al.  Methods to test visual attention online. , 2015, Journal of visualized experiments : JoVE.

[89]  Aleksandrs Slivkins,et al.  Incentivizing high quality crowdwork , 2015, SECO.

[90]  Tara S. Behrend,et al.  The viability of crowdsourcing for survey research , 2011, Behavior research methods.

[91]  Adam J. Berinsky,et al.  Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk , 2012, Political Analysis.

[92]  G. Miller,et al.  Science Perspectives on Psychological the Smartphone Psychology Manifesto on Behalf Of: Association for Psychological Science the Smartphone Psychology Manifesto Previous Research Using Mobile Electronic Devices What Smartphones Can Do Now and Will Be Able to Do in the near Future , 2022 .

[93]  Brent Simpson,et al.  Emotional reactions to losing explain gender differences in entering a risky lottery , 2010, Judgment and Decision Making.

[94]  D. Wechsler,et al.  Wechsler Adult Intelligence Scale - fourth edition , 2012 .

[95]  Ophelia Deroy,et al.  Fast lemons and sour boulders: Testing crossmodal correspondences using an internet-based testing methodology , 2013, i-Perception.

[96]  Richard R Plant,et al.  Millisecond precision psychological research in a world of commodity computers: New hardware, new problems? , 2009, Behavior research methods.

[97]  Winter A. Mason,et al.  Internet research in psychology. , 2015, Annual review of psychology.

[98]  K. Nakayama,et al.  The Cambridge Face Memory Test: Results for neurologically intact individuals and an investigation of its validity using inverted face stimuli and prosopagnosic participants , 2006, Neuropsychologia.

[99]  Benjamin A. Motz,et al.  Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task , 2015, Behavior Research Methods.

[100]  Michael E. R. Nicholls,et al.  Some participants may be better than others: Sustained attention and motivation are higher early in semester , 2015, Quarterly journal of experimental psychology.

[101]  Kenneth I Forster,et al.  DMDX: A Windows display program with millisecond accuracy , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.