A Permutation-Based Model for Crowd Labeling: Optimal Estimation and Robustness

The aggregation and denoising of crowd-labeled data is a task that has gained increased significance with the advent of crowdsourcing platforms and massive datasets. In this paper, we propose a permutation-based model for crowd labeled data that is a significant generalization of the common Dawid-Skene model, and introduce a new error metric by which to compare different estimators. Working in a high-dimensional non-asymptotic framework that allows both the number of workers and tasks to scale, we derive minimax rates of convergence for the permutation-based model that are optimal (up to logarithmic factors). We show that the permutation-based model offers significant robustness in estimation due to its richness, while surprisingly incurring only a small additional statistical penalty as compared to the Dawid-Skene model. We then design a computationally-efficient method, called the OBI-WAN estimator, that is optimal over a class intermediate between the permutation-based and the Dawid-Skene models (up to logarithmic factors), and also simultaneously achieves non-trivial guarantees over the entire permutation-based model class. Finally, we conduct synthetic simulations and experiments on real-world crowdsourcing data, and these corroborate our theoretical findings.