Crowdsourcing for Cognitive Science – The Utility of Smartphones

By 2015, there will be an estimated two billion smartphone users worldwide. This technology presents exciting opportunities for cognitive science as a medium for rapid, large-scale experimentation and data collection. At present, cost and logistics limit most study populations to small samples, restricting the experimental questions that can be addressed. In this study we investigated whether the mass collection of experimental data using smartphone technology is valid, given the variability of data collection outside of a laboratory setting. We presented four classic experimental paradigms as short games, available as a free app and over the first month 20,800 users submitted data. We found that the large sample size vastly outweighed the noise inherent in collecting data outside a controlled laboratory setting, and show that for all four games canonical results were reproduced. For the first time, we provide experimental validation for the use of smartphones for data collection in cognitive science, which can lead to the collection of richer data sets and a significant cost reduction as well as provide an opportunity for efficient phenotypic screening of large populations.

[1]  G. Logan,et al.  On the ability to inhibit simple and choice reaction time responses: a model and a method. , 1984, Journal of experimental psychology. Human perception and performance.

[2]  K L Shapiro,et al.  Temporary suppression of visual processing in an RSVP task: an attentional blink? . , 1992, Journal of experimental psychology. Human perception and performance.

[3]  L Hasher,et al.  Inhibitory attentional mechanisms and aging. , 1994, Psychology and aging.

[4]  M. Potter,et al.  A two-stage model for multiple target detection in rapid serial visual presentation. , 1995, Journal of experimental psychology. Human perception and performance.

[5]  M C Potter,et al.  Two attentional deficits in serial target search: the visual attentional blink and an amodal task-switch deficit. , 1998, Journal of experimental psychology. Learning, memory, and cognition.

[6]  G. Logan,et al.  Development of inhibitory control across the life span. , 1999, Developmental psychology.

[7]  Gordon D Logan,et al.  Horse-race model simulations of the stop-signal procedure. , 2003, Acta psychologica.

[8]  R. Thomson,et al.  A systematic review of cognitive decline in the general elderly population , 2003, International journal of geriatric psychiatry.

[9]  M. Birnbaum.  Human research and data collection via the internet. , 2004, Annual review of psychology.

[10]  Erratum: Why pictures look right when viewed from the wrong place , 2005, Nature Neuroscience.

[11]  Jeffrey W. Cooney,et al.  Top-down suppression deficit underlies working memory impairment in normal aging , 2005, Nature Neuroscience.

[12]  A. Aron,et al.  Stop the Presses , 2008, Psychological science.

[13]  Manuel Blum,et al.  reCAPTCHA: Human-Based Character Recognition via Web Security Measures , 2008, Science.

[14]  G. Logan,et al.  Response inhibition in the stop-signal paradigm , 2008, Trends in Cognitive Sciences.

[15]  Winston D. Byblow,et al.  Stop and Go: The Neural Basis of Selective Movement Prevention , 2009, Journal of Cognitive Neuroscience.

[16]  P. Rabbitt,et al.  Further analyses of the effects of practice, dropout, sex, socio-economic advantage, and recruitment cohort differences during the University of Manchester longitudinal study of cognitive change in old age , 2009, Quarterly journal of experimental psychology.

[17]  Colin Camerer,et al.  Thinking like a trader selectively reduces individuals' loss aversion , 2009, Proceedings of the National Academy of Sciences.

[18]  Peter Bakhirev,et al.  Beginning iPhone Games Development , 2010 .

[19]  D. Clery.  Galaxy evolution. Galaxy zoo volunteers share pain and glory of research. , 2011, Science.

[20]  Jon Sprouse.  A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory , 2010, Behavior research methods.

[21]  David Baker,et al.  Algorithm discovery by protein folding game players , 2011, Proceedings of the National Academy of Sciences.

[22]  J. Ziegler,et al.  Smart Phone, Smart Science: How the Use of Smartphones Can Revolutionize Research in Cognitive Science , 2011, PloS one.

[23]  Erik M. Buck.  Learning OpenGL ES for iOS: A Hands-on Guide to Modern 3D Graphics Programming , 2012 .

[24]  Todd M. Gureckis,et al.  CUNY Academic , 2016 .

[25]  S. Mourato,et al.  Happiness is greater in natural environments , 2013 .

[26]  A. Henik,et al.  Individual but not fragile: Individual differences in task control predict Stroop facilitation , 2013, Consciousness and Cognition.

[27]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[28]  Rick A Adams,et al.  Correction: Crowdsourcing for Cognitive Science – The Utility of Smartphones , 2014, PLoS ONE.

[29]  Fiona McNab,et al.  Dissociating Distractor-Filtering at Encoding and During Maintenance , 2014, Journal of experimental psychology. Human perception and performance.

[30]  Thomas H. B. FitzGerald,et al.  Transcranial Direct Current Stimulation of Right Dorsolateral Prefrontal Cortex Does Not Affect Model-Based or Model-Free Reinforcement Learning in Humans , 2014, PloS one.