Swapping bricks for clicks: Crowdsourcing longitudinal data on Amazon Turk

Locating reliable sources of generalizable longitudinal data is an extremely important issue for business research. The aim of this paper was to empirically verify that crowdsourcing can be used to source longitudinal samples. Specifically, three studies assess reliability of the Amazon Mechanical Turk Marketplace (MTurk). All three studies demonstrate that MTurk is a reliable, inexpensive source for generalizable longitudinal data. Study 1 (n=752) examines the two-month re-response rate (study 1, n=752; 75%) of a US MTurk sample. Study 2 (n=373) investigates the four- and eight-month re-response rate (56 and 38%, respectively) of a US immigrant sample. Study 3 examines the thirteen-month re-response rate (47%). Each study demonstrates minimal non-response biases and longitudinal response consistency, in terms of both demographics and personality traits. This study also independently verifies the accuracy of self-report state of residence for 94% of the participants.

[1]  David G. Rand,et al.  The promise of Mechanical Turk: how online labor markets can help theorists run behavioral experiments. , 2012, Journal of theoretical biology.

[2]  Moez Limayem,et al.  Data Collection in the Digital Age: Innovative Alterantives to Student Samples , 2014, MIS Q..

[3]  Viswanath Venkatesh,et al.  Expectation Confirmation in Information Systems Research: A Test of Six Competing Models , 2014, MIS Q..

[4]  Michael Boyer O'Leary,et al.  Beyond Being There: The Symbolic Role of Communication and Identification in Perceptions of Proximity to Geographically Dispersed Colleagues , 2014, MIS Q..

[5]  Scott B. MacKenzie,et al.  Construct Measurement and Validation Procedures in MIS and Behavioral Research: Integrating New and Existing Techniques , 2011, MIS Q..

[6]  Joel E. Collier,et al.  Self-service delight: Exploring the hedonic aspects of self-service , 2015 .

[7]  L. Xia,et al.  For our valued customers only: Examining consumer responses to preferential treatment practices , 2014 .

[8]  Jon Sprouse.  A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory , 2010, Behavior research methods.

[9]  Antonio Terracciano,et al.  Intra-individual Change in Personality Stability and Age. , 2010, Journal of research in personality.

[10]  Viswanath Venkatesh,et al.  Consumer Acceptance and Use of Information Technology: Extending the Unified Theory of Acceptance and Use of Technology , 2012, MIS Q..

[11]  Delroy L. Paulhus,et al.  The role of impulsivity in the Dark Triad of personality , 2011 .

[12]  Adam D. Galinsky,et al.  Power and overconfident decision-making , 2012 .

[13]  Panagiotis G. Ipeirotis.  Demographics of Mechanical Turk , 2010 .

[14]  Daniel J. Veit,et al.  More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk , 2011, AMCIS.

[15]  Barbara Ross Wooldridge,et al.  A new dualistic approach to brand passion: Harmonious and obsessive , 2014 .

[16]  John Dawes,et al.  Has brand loyalty declined? A longitudinal analysis of repeat purchase behavior in the UK and the USA , 2015 .

[17]  Christopher J. Holden,et al.  Assessing the reliability of the M5-120 on Amazon's mechanical Turk , 2013, Comput. Hum. Behav..

[18]  Joshua J. Jackson,et al.  Not all conscientiousness scales change alike: a multimethod, multisample study of age differences in the facets of conscientiousness. , 2009, Journal of personality and social psychology.

[19]  Paul A. Pavlou,et al.  Understanding and Mitigating Uncertainty in Online Exchange Relationships: A Principal-Agent Perspective , 2007, MIS Q..

[20]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[21]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[22]  Lin Qiu,et al.  You are what you tweet: Personality expression and perception on Twitter , 2012 .

[23]  Jesse Chandler,et al.  Using Mechanical Turk to Study Clinical Populations , 2013 .

[24]  Merrill Warkentin,et al.  Fear Appeals and Information Security Behaviors: An Empirical Study , 2010, MIS Q..

[25]  Joseph M. Paxton,et al.  Reflection and Reasoning in Moral Judgment , 2022 .

[26]  Anol Bhattacherjee,et al.  Understanding Changes in Belief and Attitude Toward Information Technology Usage: A Theoretical Model and Longitudinal Test , 2004, MIS Q..

[27]  Brian J. Baldus,et al.  Online brand community engagement: Scale development and validation , 2015 .

[28]  Dionysis Skarmeas,et al.  When consumers doubt, Watch out! The role of CSR skepticism , 2013 .

[29]  Amar Cheema,et al.  Data collection in a flat world: the strengths and weaknesses of mechanical turk samples , 2013 .

[30]  C. Sibley,et al.  The Mini-IPIP6: tiny yet highly stable markers of Big Six personality , 2013 .

[31]  Ingwer Borg,et al.  The measurement equivalence of Big Five factor markers for persons with different levels of education. , 2010, Journal of research in personality.

[32]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[33]  Elizabeth M. Poposki,et al.  Detecting and Deterring Insufficient Effort Responding to Surveys , 2012 .