Predicting QoS in Scheduled Crowdsourcing

Crowdsourcing has emerged as a new paradigm for outsourcing simple for humans yet hard to automate tasks to an undefined network of people. Crowdsourcing platforms like Amazon Mechanical Turk provide scalability and flexibility for customers that need to get manifold similar independent jobs done. However, such platforms do not provide certain guarantees for their services regarding the expected job quality and the time of processing, although such guarantees are advantageous from the perspective of Business Process Management. In this paper, we consider an alternative architecture of a crowdsourcing platform, where the workers are assigned to tasks by the platform according to their availability and skills. We propose the technique for estimating accomplishable guarantees and negotiating Service Level Agreements in such an environment.

[1]  Schahram Dustdar,et al.  Stimulating Skill Evolution in Market-Based Crowdsourcing , 2011, BPM.

[2]  Sudhir Agarwal,et al.  Managing Quality of Human-Based eServices , 2008, ICSOC Workshops.

[3]  Daren C. Brabham.  Crowdsourcing as a model for problem solving: Leveraging the collective intelligence of online communities for public good , 2010 .

[4]  Panagiotis G. Ipeirotis.  Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[5]  Alon Y. Halevy,et al.  Crowdsourcing systems on the World-Wide Web , 2011, Commun. ACM.

[6]  Isabel Ramos,et al.  The past, present and future of social networking and outsourcing: impact on theory and practice , 2010 .

[7]  Ehud D. Karnin,et al.  Crowdsourcing in the Document Processing Practice - (A Short Practitioner/Visionary Paper) , 2010, ICWE Workshops.

[8]  Hans Thies,et al.  Statistical Quality Control for Human-Based Electronic Services , 2010, ICSOC.

[9]  Panagiotis G. Ipeirotis.  Demographics of Mechanical Turk , 2010 .

[10]  Schahram Dustdar,et al.  Resource and Agreement Management in Dynamic Crowdcomputing Environments , 2011, 2011 IEEE 15th International Enterprise Distributed Object Computing Conference.

[11]  Schahram Dustdar,et al.  QoS-Based Task Scheduling in Crowdsourcing Environments , 2011, ICSOC.

[12]  Michael Pinedo,et al.  Scheduling: Theory, Algorithms, and Systems , 1994 .

[13]  Paolo Toth,et al.  Models and algorithms for a staff scheduling problem , 2003, Math. Program..

[14]  Carl Adams.  Managing crowdsourcing assignments , 2011 .

[15]  Florian Daniel,et al.  Current Trends in Web Engineering , 2010, Lecture Notes in Computer Science.

[16]  Marco Aiello,et al.  Deriving business processes with service level agreements from early requirements , 2011, J. Syst. Softw..