Answer Aggregation for Crowdsourcing Microtasks using Approximate Global Optimal Searching

In micro-task crowdsourcing, the answers collected from the crowd are ambiguous and the final answer aggregation is challenging due to the various capabilities and knowledge background of the voluntary participants on the Internet. In this paper, we extend the local optimal result of Expectation-Maximization(EM) approach and propose an approximate global optimal algorithm for answer aggregation of crowdsourcing microtasks with binary answers. Our algorithm is expected to improve the accuracy of real answer estimation through further likelihood maximization. We conduct extensive experiments on both simulated and real-world datasets, and the experimental results illustrate that the proposed approach can obtain better estimation results and has higher performance than regular EM-based algorithms.

[1]  Minyi Guo,et al.  MELODY: A Long-Term Dynamic Quality-Aware Incentive Mechanism for Crowdsourcing , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[2]  Jennifer Widom,et al.  Towards Globally Optimal Crowdsourcing Quality Management: The Uniform Worker Setting , 2016, SIGMOD Conference.

[3]  Devavrat Shah,et al.  Efficient crowdsourcing for multi-class labeling , 2013, SIGMETRICS '13.

[4]  Anirban Dasgupta,et al.  Aggregating crowdsourced binary ratings , 2013, WWW.

[5]  Reynold Cheng,et al.  QASCA: A Quality-Aware Task Assignment System for Crowdsourcing Applications , 2015, SIGMOD Conference.

[6]  Lei Chen,et al.  Whom to Ask? Jury Selection for Decision Making Tasks on Micro-blog Services , 2012, Proc. VLDB Endow..

[7]  Zhifeng Bao,et al.  Crowdsourced POI labelling: Location-aware result inference and Task Assignment , 2016, 2016 IEEE 32nd International Conference on Data Engineering (ICDE).

[8]  Bo Pang,et al.  Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales , 2005, ACL.

[9]  George Kesidis,et al.  Multicategory Crowdsourcing Accounting for Variable Task Difficulty, Worker Skill, and Worker Intention , 2015, IEEE Transactions on Knowledge and Data Engineering.

[10]  Beng Chin Ooi,et al.  CDAS: A Crowdsourcing Data Analytics System , 2012, Proc. VLDB Endow..

[11]  Tim Kraska,et al.  Leveraging transitive relations for crowdsourced joins , 2013, SIGMOD '13.

[12]  Heng Ji,et al.  FaitCrowd: Fine Grained Truth Discovery for Crowdsourced Data Aggregation , 2015, KDD.

[13]  Xi Chen,et al.  Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing , 2014, J. Mach. Learn. Res..

[14]  Aditya G. Parameswaran,et al.  So who won?: dynamic max discovery with the crowd , 2012, SIGMOD Conference.

[15]  Barbara Carminati,et al.  CrowdEval: A Cost-Efficient Strategy to Evaluate Crowdsourced Worker's Reliability , 2018, AAMAS.

[16]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.