CONAN: A framework for detecting and handling collusion in crowdsourcing

Abstract In contrast to the traditional view that individuals should work independently to realize the crowd wisdom, crowdsourcing workers often collaborate with each other in task processing either explicitly or implicitly. Some may even collude for obtaining rewards easily, for example, by plagiarizing others’ answers. Collusion behavior sabotages the independency among workers, and will subvert the benefits of task redundancy that is commonly adopted in crowdsourcing. Therefore, dealing with collusion is critical for ensuring the quality of crowdsourcing. Existing work usually treats all the collusive answers as being harmful, thus simply filters away them once they are detected. However, it is not always the best strategy in practice. In particular, when the collusive answers are plagiarized from a worker with good ability, utilizing them instead of simple elimination can benefit the result quality. In this work, we first propose a collusion-aware framework for detecting and handling collusion in crowdsourcing properly. Second, we design a collusion detection method based on the statistical test of the consistency of workers’ answers across tasks. Third, we provide a theoretical means to determine when collusive answers should be kept and utilized, then we design a collusion-aware answer aggregation method. Finally, we conducted thorough evaluation with both synthetic and real-world datasets, and the results demonstrate the effectiveness of our approach.