An Explorative Approach for Crowdsourcing Tasks Design

Crowdsourcing applications are becoming widespread; they cover very different scenarios, including opinion mining, multimedia data annotation, localised information gathering, marketing campaigns, expert response gathering, and so on. The quality of the outcome of these applications depends on different design parameters and constraints, and it is very hard to judge about their combined effects without doing some experiments; on the other hand, there are no experiences or guidelines that tell how to conduct experiments, and thus these are often conducted in an ad-hoc manner, typically through adjustments of an initial strategy that may converge to a parameter setting which is quite different from the best possible one. In this paper we propose a comparative, explorative approach for designing crowdsourcing tasks. The method consists of defining a representative set of execution strategies, then execute them on a small dataset, then collect quality measures for each candidate strategy, and finally decide the strategy to be used with the complete dataset.

[1]  Shigeo Matsubara,et al.  Efficient Task Decomposition in Crowdsourcing , 2014, PRIMA.

[2]  Andrew McGregor,et al.  AutoMan: a platform for integrating human-based and digital computation , 2012, OOPSLA '12.

[3]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[4]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[5]  Rob Miller,et al.  Crowdsourced Databases: Query Processing with People , 2011, CIDR.

[6]  Lydia B. Chilton,et al.  TurKit: Tools for iterative tasks on mechanical turk , 2009, 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[7]  Lei Chen,et al.  MaC: A Probabilistic Framework for Query Answering with Machine-Crowd Collaboration , 2014, CIKM.

[8]  Hao Wu,et al.  Relationship between quality and payment in crowdsourced design , 2014, Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD).

[9]  Elena Paslaru Bontas Simperl,et al.  Making your semantic application addictive: incentivizing users! , 2012, WIMS '12.

[10]  Praveen Paritosh,et al.  The anatomy of a large-scale human computation engine , 2010, HCOMP '10.

[11]  Hamid R. Rabiee,et al.  A unified statistical framework for crowd labeling , 2015, Knowledge and Information Systems.

[12]  A. P. deVries,et al.  How Crowdsourcable is Your Task , 2011 .

[13]  Alessandro Bozzon,et al.  Answering search queries with CrowdSearcher , 2012, WWW.

[14]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[15]  Alessandro Bozzon,et al.  Reactive crowdsourcing , 2013, WWW.

[16]  Matthew Lease,et al.  Inferring missing relevance judgments from crowd workers via probabilistic matrix factorization , 2012, SIGIR '12.

[17]  Tom Minka,et al.  How To Grade a Test Without Knowing the Answers - A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing , 2012, ICML.

[18]  Lydia B. Chilton,et al.  Exploring iterative and parallel human computation processes , 2010, HCOMP '10.

[19]  Aniket Kittur,et al.  CrowdWeaver: visually managing complex crowd work , 2012, CSCW.

[20]  Alexis Battle,et al.  The jabberwocky programming environment for structured social computing , 2011, UIST.

[21]  Abraham Bernstein,et al.  How to translate a book within an hour: towards general purpose programmable human computers with CrowdLang , 2012, WebSci '12.

[22]  Milad Shokouhi,et al.  Community-based bayesian aggregation models for crowdsourcing , 2014, WWW.

[23]  Benjamin Satzger,et al.  Crowdsourcing tasks to social networks in BPEL4People , 2012, World Wide Web.

[24]  Tim Kraska,et al.  CrowdDB: answering queries with crowdsourcing , 2011, SIGMOD '11.