CONSENSUS-BASED CROWDSOURCING: TECHNIQUES AND APPLICATIONS

Crowdsourcing solutions are receiving more and more attention in the recent literature about social computing and distributed problem solving. In general terms, crowdsourcing can be considered as a social-computing model aimed at fostering the autonomous formation and emergence of the so-called wisdom of the crowd. Quality assessment is a crucial issue for the effectiveness of crowdsourcing systems, both for what concerns task and worker management. Another aspect to be considered in crowdsourcing systems is about the kind of contributions workers can make. Usually, crowdsourcing approaches rely only on tasks where workers have to decide among a predefined set of possible solutions. On the other hand, tasks leaving the workers a higher level of freedom in producing their answer (e.g., free-hand drawing) are more difficult to be managed and verified. In the Thesis, we present the LiquidCrowd approach based on consensus and trustworthiness techniques for managing the execution of collaborative tasks. By collaborative task, we refer to a task for which a factual answer is not possible/appropriate, or a task whose result depends on the personal perception/point-of-view of the worker. We introduce the notion of worker trustworthiness to denote the worker “reliability”, namely her/his capability to foster the successful completion of tasks. Furthermore, we improve the conventional score-based mechanism by introducing the notion of award that is a bonus provided to those workers that contribute to reach the consensus within groups. This way, groups with certain trustworthiness requirements can be composed on-demand, to deal with complex tasks, like for example tasks where consensus has not been reached during the first execution. In LiquidCrowd, we define a democratic mechanism based on the notion of supermajority to enable the flexible specification of the expected degree of agreement required for obtaining the consensus within a worker group. In LiquidCrowd, three task typologies are provided: choice, where the worker is asked to choose the answer among a list of predefined options; range, where the worker is asked to provide a free-numeric answer; proposition, where the worker is asked to provide a free text answer. To evaluate the quality of the produced results obtained through LiquidCrowd consensus techniques, we perform a testing against the SQUARE crowdsourcing benchmark. Furthermore, to evaluate the capability of LiquidCrowd to effectively support a real problem, real case studies about web data classification have been selected.