Crowdsourcing for Survey Research : where Amazon Mechanical Turks deviates from conventional survey methods

Information systems research has started to use crowdsourcing platforms such as Amazon Mechanical Turks (MTurk) for scientific research, recently. In particular, MTurk provides a scalable, cheap work-force that can also be used as a pool of potential respondents for online survey research. In light of the increasing use of crowdsourcing platforms for survey research, the authors aim to contribute to the understanding of its appropriate usage. Therefore, they assess if samples drawn from MTurk deviate from those drawn via conventional online surveys (COS) in terms of answers in relation to relevant e-commerce variables and test the data in a nomological network for assessing differences in effects. The authors compare responses from 138 MTurk workers with those of 150 German shoppers recruited via COS. The findings indicate, inter alia, that MTurk workers tend to exhibit more positive word-of mouth, perceived risk, customer orientation and commitment to the focal company. The authors discuss the study- results, point to limitations, and provide avenues for further research.

[1]  Christopher L. Tucci,et al.  Crowdsourcing As a Solution to Distant Search , 2012 .

[2]  Gianfranco Walsh,et al.  A problem-solving perspective on governance and product design in open source software projects: Conceptual issues and exploratory evidence , 2013, ICIS.

[3]  Lorrie Faith Cranor,et al.  Are your participants gaming the system?: screening mechanical turk workers , 2010, CHI.

[4]  S. Müller,et al.  Combined Non-Linear Pose Estimation from Points and Lines , 2011 .

[5]  Gianfranco Walsh,et al.  Customer-based corporate reputation of a service firm: scale development and validation , 2007 .

[6]  Petra Schubert,et al.  Cloud Computing for Standard ERP Systems: Reference Framework and Research Agenda , 2011 .

[7]  Paul A. Pavlou,et al.  Evidence of the Effect of Trust Building Technology in Electronic Markets: Price Premiums and Buyer Behavior , 2002, MIS Q..

[8]  J. Steenkamp,et al.  Assessing Measurement Invariance in Cross-National Consumer Research , 1998 .

[9]  Katja Hutter,et al.  Crowdsourcing for Goodness Sake: Impact of Incentive Preference on Contribution Behavior for Social Innovation , 2012 .

[10]  Richard P. Bagozzi,et al.  Specification, evaluation, and interpretation of structural equation models , 2012 .

[11]  Jennifer Preece,et al.  Electronic Survey Methodology: A Case Study in Reaching Hard-to-Involve Internet Users , 2003, Int. J. Hum. Comput. Interact..

[12]  A. Acquisti,et al.  Reputation as a sufficient condition for data quality on Amazon Mechanical Turk , 2013, Behavior Research Methods.

[14]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[15]  Kevin Crowston,et al.  Amazon Mechanical Turk: A Research Tool for Organizations and Information Systems Scholars , 2012, Shaping the Future of ICT Research.

[16]  Jan Marco Leimeister,et al.  The Effect of Rating Scales on Decision Quality and User Attitudes in Online Innovation Communities , 2012, Int. J. Electron. Commer..

[17]  C. Fornell,et al.  Evaluating structural equation models with unobservable variables and measurement error. , 1981 .

[18]  L. F. Cunningham,et al.  Perceived risk and the consumer buying process: internet airline reservations , 2005 .

[19]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[20]  Ansgar Scherp,et al.  Integrating Multimedia Metadata Standarts and Metadata Formats with the Multimedia Metadata Ontology: Method and Examples , 2012 .

[21]  Seounmi Youn,et al.  Online Word-of-Mouth (or Mouse): An Exploration of Its Antecedents and Consequences , 2006, J. Comput. Mediat. Commun..

[22]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[23]  Dwayne D. Gremler,et al.  Understanding Relationship Marketing Outcomes , 2002 .

[24]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[25]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[26]  Daniel J. Veit,et al.  More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk , 2011, AMCIS.

[27]  Klaus G. Troitzsch,et al.  Towards a methodology of developing models of e-service quality assessment in healthcare , 2011 .

[28]  Sung-joon Yoon,et al.  The antecedents and consequences of trust in online-purchase decisions , 2002 .

[29]  Steffen Staab,et al.  Mobile Facets - faceted search and exploration of open social media data on a touchscreen mobile phone , 2011 .

[30]  G. Hofstede.  The Cultural Relativity of Organizational Practices and Theories , 1983 .

[31]  P. M. Podsakoff,et al.  Self-Reports in Organizational Research: Problems and Prospects , 1986 .

[32]  Matthew Lease,et al.  Crowdsourcing 101: putting the WSDM of crowds to work for you , 2011, WSDM '11.

[33]  Marios Koufaris,et al.  The development of initial trust in an online company by new customers , 2004, Inf. Manag..

[34]  Michael T. Brannick,et al.  Critical comments on applying covariance structure modeling , 1995 .

[35]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[36]  Gianfranco Walsh,et al.  The customer-based corporate reputation scale: replication and short form , 2009 .

[37]  Dwayne D. Gremler,et al.  Electronic word-of-mouth via consumer-opinion platforms: What motivates consumers to articulate themselves on the Internet? , 2004 .

[38]  M. Lindell,et al.  Accounting for common method variance in cross-sectional research designs. , 2001, The Journal of applied psychology.

[39]  Jan Marco Leimeister,et al.  Does Collaboration among Participants Lead to Better Ideas in IT-Based Idea Competitions? An Empirical Investigation , 2010, HICSS.

[40]  Yulin Fang,et al.  Understanding online customer repurchasing intention and the mediating role of trust – an empirical investigation in two developed countries , 2009, Eur. J. Inf. Syst..

[41]  D. Bromley.  Relationships between personal and corporate reputation , 2001 .

[42]  Jan Marco Leimeister,et al.  Leveraging Crowdsourcing: Activation-Supporting Components for IT-Based Ideas Competition , 2009, J. Manag. Inf. Syst..

[43]  Deepak Khazanchi,et al.  Crowdsourcing Typology: A Review of IS Research and Organizations , 2014 .

[44]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[45]  Mary P. Harrison,et al.  Understanding the Relationships Between Commitment and Voice , 2012 .

[46]  M. Laroche,et al.  Internet versus Bricks-and-Mortar Retailers: An Investigation into Intangibility and Its Consequences , 2005 .

[47]  C. Fombrun,et al.  What's in a Name? Reputation Building and Corporate Strategy , 1990 .

[48]  David L. Mothersbaugh,et al.  The Positive and Negative Effects of Switching Costs on Relational Outcomes , 2007 .

[49]  P. Franses,et al.  The effect of relational constructs on customer referrals and number of services purchased from a multiservice provider: Does age of relationship matter? , 2002 .

[50]  Kurt Matzler,et al.  Communitition: The Tension between Competition and Collaboration in Community-Based Design Contests , 2011 .

[51]  B. Byrne.  A Primer of Lisrel: Basic Applications and Programming for Confirmatory Factor Analytic Models , 1989 .

[52]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[53]  Ann Majchrzak,et al.  Towards an Information Systems Perspective and Research Agenda on Crowdsourcing for Innovation , 2013, J. Strateg. Inf. Syst..

[54]  Malte Knauf,et al.  Explicit and Implicit Schema Information on the Linked Open Data Cloud: Joined Forces or Antagonists? , 2012 .

[55]  Adam J. Berinsky,et al.  Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk , 2012, Political Analysis.

[56]  K. Bräunlich,et al.  Der neue Personalausweis zur Authentifizierung von Wählern bei Onlinewahlen , 2011 .