Exploiting Structure in Crowdsourcing Tasks via Latent Factor Models

Internet crowdsourcing services such as the Amazon Mechanical Turk (1) and the ESP Game (15) have become important tools for the machine learning community by facilitating the distributed labeling of large datasets at little cost. A key challenge when using crowdsourcing to label databases is the need to derive high quality labels by aggregating the responses from labelers of varying reliability over data instances of varying difficulty. Existing algorithms for quality control and label inference (14; 17; 10) suffer several significant shortcomings: (1) Existing methods are incapable of modeling interaction effects between labeler and data items, such as when some labelers have specialized knowledge about a particular subset of items. (2) Existing algorithms assume that labelers’ accuracies, as well as data instances’ difficulties, are independent. In reality, there may be a priori information about labelers (or data instances) that predicts those labelers’ accuracy at the labeling task. Analogously, certain features shared among data instances may predict their difficulty of being labeled correctly. In this paper, we present an algorithm that addresses both of these shortcomings. We demonstrate that the proposed algorithm delivers superior accuracy, compared to previous methods, of inferring data labels on a difficult facial expression labeling task. Finally, we show that our proposed model subsumes certain previous models as special cases.