Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys

Crowdsourcing is increasingly being used as a means to tackle problems requiring human intelligence. With the ever-growing worker base that aims to complete microtasks on crowdsourcing platforms in exchange for financial gains, there is a need for stringent mechanisms to prevent exploitation of deployed tasks. Quality control mechanisms need to accommodate a diverse pool of workers, exhibiting a wide range of behavior. A pivotal step towards fraud-proof task design is understanding the behavioral patterns of microtask workers. In this paper, we analyze the prevalent malicious activity on crowdsourcing platforms and study the behavior exhibited by trustworthy and untrustworthy workers, particularly on crowdsourced surveys. Based on our analysis of the typical malicious activity, we define and identify different types of workers in the crowd, propose a method to measure malicious activity, and finally present guidelines for the efficient design of crowdsourced surveys.

[1]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[2]  Juliet M. Corbin,et al.  Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory , 2008 .

[3]  Nicolas Tsapatsoulis,et al.  Automatic annotation of image databases based on implicit crowdsourcing, visual concept modeling and evolution , 2012, Multimedia Tools and Applications.

[4]  Tara S. Behrend,et al.  The viability of crowdsourcing for survey research , 2011, Behavior research methods.

[5]  Lukas Biewald,et al.  Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing , 2011, Human Computation.

[6]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.

[7]  Gabriella Kazai,et al.  Worker types and personality traits in crowdsourcing relevance labels , 2011, CIKM '11.

[8]  Hisashi Kashima,et al.  Leveraging Crowdsourcing to Detect Improper Tasks in Crowdsourcing Marketplaces , 2013, IAAI.

[9]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[10]  Daren C. Brabham.  Crowdsourcing as a model for problem solving: Leveraging the collective intelligence of online communities for public good , 2010 .

[11]  Gang Wang,et al.  Serf and turf: crowdturfing for fun and profit , 2011, WWW.

[12]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[13]  Carla Simone,et al.  Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work Companion , 2012, CSCW 2012.

[14]  Kwong-Sak Leung,et al.  A Survey of Crowdsourcing Systems , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[15]  Gianluca Demartini,et al.  Mechanical Cheat: Spamming Schemes and Adversarial Techniques on Crowdsourcing Platforms , 2012, CrowdSearch.

[16]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[17]  Robert L. Glass,et al.  Contemporary Application-Domain Taxonomies , 1995, IEEE Softw..

[18]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[19]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[20]  Scott R. Klemmer,et al.  Shepherding the crowd: managing and providing feedback to crowd workers , 2011, CHI Extended Abstracts.

[21]  Stefan Dietze,et al.  A taxonomy of microtasks on the web , 2014, HT.

[22]  Daniel J. Veit,et al.  More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk , 2011, AMCIS.

[23]  N. Hoffart.  Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 2000 .

[24]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[25]  Gabriella Kazai,et al.  An analysis of human factors and label accuracy in crowdsourcing relevance judgments , 2013, Information Retrieval.

[26]  Sabine Buchholz,et al.  Crowdsourcing Preference Tests, and How to Detect Cheating , 2011, INTERSPEECH.

[27]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[28]  A. P. deVries,et al.  How Crowdsourcable is Your Task , 2011 .

[29]  Arjen P. de Vries,et al.  Increasing cheat robustness of crowdsourcing tasks , 2013, Information Retrieval.

[30]  Craig Gentry,et al.  Non-interactive Verifiable Computing: Outsourcing Computation to Untrusted Workers , 2010, CRYPTO.

[31]  Marco Brambilla,et al.  A revenue sharing mechanism for federated search and advertising , 2012, WWW.

[32]  Frank M. Shipman,et al.  Experiences surveying the crowd: reflections on methods, participation, and reliability , 2013, WebSci.