The Complexity of Crowdsourcing: Theoretical Problems in Human Computation

What does theoretical computer science have to say about human computation? We identify three problems at the intersection of crowdsourcing, operations research, and theoretical computer science whose solution would have a major impact on the design, evaluation, and construction of real crowdsourcing systems. In some cases, these problems can let us sidestep apparently difficult HCI challenges by making appropriate choices at the algorithmic level. In other contexts, theoretical tools provide a formal basis for evaluating the performance of algorithms and classifying the difficulty of tasks in crowdsourcing. Our problems are illustrated through two recent projects. The first, Turkomatic, is an attempt to construct a “universal” algorithm for generating workflows on microtask crowdsourcing platforms. The second, MobileWorks, is a new crowdsourcing engine designed from the bottom up to provide appropriate abstractions between the theoretical elements of human computation systems and interface/design questions. It is hoped that this analysis can spur the development of theoretical frameworks for understanding algorithms involving human computation.