Detecting malicious campaigns in crowdsourcing platforms

Crowdsourcing systems enable new opportunities for requesters with limited funds to accomplish various tasks using human computation. However, the power of human computation is abused by malicious requesters who create malicious campaigns to manipulate information in web systems such as social networking sites, online review sites, and search engines. To mitigate the impact and reach of these malicious campaigns to targeted sites, we propose and evaluate a machine learning based classification approach for detecting malicious campaigns in crowdsourcing platforms as a first line of defense. Specifically, we (i) conduct a comprehensive analysis to understand the characteristics of malicious campaigns and legitimate campaigns in crowdsourcing platforms, (ii) propose various features to distinguish between malicious campaigns and legitimate campaigns, and (iii) evaluate a classification approach against baselines. Our experimental results show that our proposed approaches effectively detect malicious campaigns with low false negative and false positive rates.

[1]  Vern Paxson,et al.  Trafficking Fraudulent Accounts: The Role of the Underground Market in Twitter Spam and Abuse , 2013, USENIX Security Symposium.

[2]  Hui-Huang Hsu,et al.  Feature Selection via Correlation Coefficient Clustering , 2010, J. Softw..

[3]  James Caverlee,et al.  Who is the barbecue king of texas?: a geo-spatial approach to finding local experts on twitter , 2014, SIGIR.

[4]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[5]  Gang Wang,et al.  Serf and turf: crowdturfing for fun and profit , 2011, WWW.

[6]  Wilfried N. Gansterer,et al.  On the Relationship Between Feature Selection and Classification Accuracy , 2008, FSDM.

[7]  Anna Cinzia Squicciarini,et al.  Uncovering Crowdsourced Manipulation of Online Reviews , 2015, SIGIR.

[8]  Hisashi Kashima,et al.  Leveraging non-expert crowdsourcing workers for improper task detection in crowdsourcing marketplaces , 2014, Expert Syst. Appl..

[9]  Roi Blanco,et al.  Machine-Learning for Spammer Detection in Crowd-Sourcing , 2012, HCOMP@AAAI.

[10]  Christopher Krügel,et al.  Framing Dependencies Introduced by Underground Commoditization , 2015, WEIS.

[11]  Lora Aroyo,et al.  Measuring crowd truth: disagreement metrics combined with worker behavior filters , 2013 .

[12]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[13]  Krishna P. Gummadi,et al.  Towards Detecting Anomalous User Behavior in Online Social Networks , 2014, USENIX Security Symposium.

[14]  Gang Wang,et al.  Social Turing Tests: Crowdsourcing Sybil Detection , 2012, NDSS.

[15]  David A. Forsyth,et al.  Utility data annotation with Amazon Mechanical Turk , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[16]  Kyumin Lee,et al.  The Dark Side of Micro-Task Marketplaces: Characterizing Fiverr and Automatically Detecting Crowdturfing , 2014, ICWSM.

[17]  Gianluca Stringhini,et al.  Poultry markets: on the underground economy of twitter followers , 2012, CCRV.

[18]  Jong Kim,et al.  CrowdTarget: Target-based Detection of Crowdturfing in Online Social Networks , 2015, CCS.

[19]  Emiliano De Cristofaro,et al.  Paying for Likes?: Understanding Facebook Like Fraud Using Honeypots , 2014, Internet Measurement Conference.

[20]  Panagiotis G. Ipeirotis,et al.  The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk , 2015, WWW.

[21]  Kyumin Lee,et al.  Crowds, Gigs, and Super Sellers: A Measurement Study of a Supply-Driven Crowdsourcing Marketplace , 2015, ICWSM.

[22]  Stefan Savage,et al.  Dirty Jobs: The Role of Freelance Labor in Web Service Abuse , 2011, USENIX Security Symposium.

[23]  Tim Kraska,et al.  CrowdDB: answering queries with crowdsourcing , 2011, SIGMOD '11.

[24]  Hector Garcia-Molina,et al.  Quality control for comparison microtasks , 2012, CrowdKDD '12.

[25]  Yiming Yang,et al.  A Comparative Study on Feature Selection in Text Categorization , 1997, ICML.

[26]  Kyumin Lee,et al.  Crowdturfers, Campaigns, and Social Media: Tracking and Revealing Crowdsourced Manipulation of Social Media , 2013, ICWSM.