A Multi-dimensional Approach to Crowd-Consensus Modeling and Evaluation

In this paper, we propose a multi-dimensional approach to support modeling and consensus management in collective crowdsourcing applications/problems. We define the notion of crowd-consensus, and, for each dimension of analysis, we set pre-defined dimensional levels capturing the different variabilities characterizing the crowd-consensus along that dimension in different applications/problems. The design of a crowdsourcing task for a given target problem requires to characterize the task with respect to each dimension following a pattern-based approach.

[1]  Efraim Turban,et al.  What can crowdsourcing do for decision support? , 2014, Decis. Support Syst..

[2]  Gianluca Demartini,et al.  ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking , 2012, WWW.

[3]  Christian Timmerer,et al.  Survey of web-based crowdsourcing frameworks for subjective quality assessment , 2014, 2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP).

[4]  Martin Schader,et al.  Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes , 2011, AMCIS.

[5]  Alessandro Bozzon,et al.  Reactive crowdsourcing , 2013, WWW.

[6]  AnHai Doan,et al.  Matching Schemas in Online Communities: A Web 2.0 Approach , 2008, 2008 IEEE 24th International Conference on Data Engineering.

[7]  Schahram Dustdar,et al.  Incentives and rewarding in social computing , 2013, CACM.

[8]  Bernardo A. Huberman,et al.  Rankr: A Mobile System for Crowdsourcing Opinions , 2011, MobiCASE.

[9]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[10]  Silvana Castano,et al.  Combining crowd consensus and user trustworthiness for managing collective tasks , 2016, Future Gener. Comput. Syst..

[11]  Mark Dredze,et al.  Annotating Named Entities in Twitter Data with Crowdsourcing , 2010, Mturk@HLT-NAACL.

[12]  Kwong-Sak Leung,et al.  A Survey of Crowdsourcing Systems , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[13]  Michael Rosemann,et al.  Crowdsourcing Information Systems - A Systems Theory Perspective , 2011 .

[14]  Alon Y. Halevy,et al.  Crowdsourcing systems on the World-Wide Web , 2011, Commun. ACM.