Crowdsourcing Task Assignment with Online Profile Learning

In this paper, we present a reference framework called Argo+ for worker-centric crowdsourcing where task assignment is characterized by feature-based representation of both tasks and workers and learning techniques are exploited to online predict the most appropriate task to execute for a requesting worker. On the task side, features are used to represent requirements expressed in terms of knowledge expertise that are asked to workers for being involved in the task execution. On the worker side, features are used to compose a profile, namely a structured description of the worker capabilities in executing tasks. Experimental results obtained on a real crowdsourcing campaign are discussed by comparing the performance of Argo+ against a baseline with conventional task assignment techniques.

[1]  Jennifer Widom,et al.  Understanding Workers, Developing Effective Tasks, and Enhancing Marketplace Dynamics: A Study of a Large Crowdsourcing Marketplace , 2017, Proc. VLDB Endow..

[2]  Guoliang Li,et al.  Truth Inference in Crowdsourcing: Is the Problem Solved? , 2017, Proc. VLDB Endow..

[3]  Stephen J. Roberts,et al.  Bayesian Methods for Intelligent Task Assignment in Crowdsourcing Systems , 2015, Decision Making.

[4]  Sihem Amer-Yahia,et al.  Motivation-Aware Task Assignment in Crowdsourcing , 2017, EDBT.

[5]  Schahram Dustdar,et al.  QoS-Based Task Scheduling in Crowdsourcing Environments , 2011, ICSOC.

[6]  Guoliang Li,et al.  Crowdsourced Data Management: A Survey , 2016, IEEE Transactions on Knowledge and Data Engineering.

[7]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[8]  Gianluca Demartini,et al.  Pick-a-crowd: tell me what you like, and i'll tell you what to do , 2013, CIDR.

[9]  M. Narasimha Murty,et al.  On Finding the Natural Number of Topics with Latent Dirichlet Allocation: Some Observations , 2010, PAKDD.

[10]  Qiang Yang,et al.  Cross-task crowdsourcing , 2013, KDD.

[11]  Silvana Castano,et al.  Combining crowd consensus and user trustworthiness for managing collective tasks , 2016, Future Gener. Comput. Syst..

[12]  Gabriella Kazai,et al.  Worker types and personality traits in crowdsourcing relevance labels , 2011, CIKM '11.

[13]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[14]  Edward Curry,et al.  A capability requirements approach for predicting worker performance in crowdsourcing , 2013, 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing.

[15]  Adam Tauman Kalai,et al.  A Crowd of Your Own: Crowdsourcing for On-Demand Personalization , 2014, HCOMP.

[16]  Beng Chin Ooi,et al.  iCrowd: An Adaptive Crowdsourcing Framework , 2015, SIGMOD Conference.

[17]  Nicholas R. Jennings,et al.  Efficient crowdsourcing of unknown experts using bounded multi-armed bandits , 2014, Artif. Intell..

[18]  Y. Narahari,et al.  A Multiarmed Bandit Incentive Mechanism for Crowdsourcing Demand Response in Smart Grids , 2014, AAAI.

[19]  Mingyan Liu,et al.  An Online Learning Approach to Improving the Quality of Crowd-Sourcing , 2015, SIGMETRICS.

[20]  Fabio Casati,et al.  Modeling, Enacting, and Integrating Custom Crowdsourcing Processes , 2015, TWEB.

[21]  Claudio Gentile,et al.  A Gang of Bandits , 2013, NIPS.

[22]  Emmanuel Müller,et al.  Discovering Multiple Clustering Solutions: Grouping Objects in Different Views of the Data , 2012, ICDE.

[23]  Sihem Amer-Yahia,et al.  Human Factors in Crowdsourcing , 2016, Proc. VLDB Endow..

[24]  Silvana Castano,et al.  A Multi-dimensional Approach to Crowd-Consensus Modeling and Evaluation , 2015, ER.

[25]  Gagan Goel,et al.  Allocating tasks to workers with matching constraints: truthful mechanisms for crowdsourcing markets , 2014, WWW.

[26]  Max Welling,et al.  Fast collapsed gibbs sampling for latent dirichlet allocation , 2008, KDD.

[27]  Ricardo Kawase,et al.  Training Workers for Improving Performance in Crowdsourcing Microtasks , 2015, EC-TEL.

[28]  Jorge Gonçalves,et al.  Task Routing and Assignment in Crowdsourcing based on Cognitive Abilities , 2017, WWW.

[29]  Sihem Amer-Yahia,et al.  Personalized and Diverse Task Composition in Crowdsourcing , 2018, IEEE Transactions on Knowledge and Data Engineering.

[30]  Devavrat Shah,et al.  Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems , 2011, Oper. Res..