Ability Grouping of Crowd Workers via Reward Discrimination

We develop a mechanism for setting discriminated reward prices in order to group crowd workers according to their abilities. Generally, a worker has a certain level of confidence in the correctness of her answers, and asking about it is useful for estimating the probability of correctness. However, we need to overcome two main obstacles to utilize confidence for inferring correct answers. One is that a worker is not always well-calibrated. Since she is sometimes over/underconfident, her confidence does not always coincide with the probability of correctness. The other is that she does not always truthfully report her confidence. Thus, we design an indirect mechanism that enables a worker to declare her confidence by choosing a desirable reward plan from the set of plans that correspond to different confidence intervals. Our mechanism ensures that choosing a plan including true confidence maximizes the worker's expected utility. We also propose a method that composes a set of plans that can achieve requester-specified accuracy in estimating the correct answer using a small number of workers. We show our experimental results using Amazon Mechanical Turk.

[1]  David C. Parkes,et al.  Peer prediction without a common prior , 2012, EC '12.

[2]  Chien-Ju Ho,et al.  Online Task Assignment in Crowdsourcing Markets , 2012, AAAI.

[3]  Ramachandra Kota,et al.  Cooperative Virtual Power Plant Formation Using Scoring Rules , 2012, AAAI.

[4]  Luis von Ahn.  Human Computation , 2008, ICDE.

[5]  Manu Sridharan,et al.  Predicting your own effort , 2012, AAMAS.

[6]  Mausam,et al.  Crowdsourcing Control: Moving Beyond Multiple Choice , 2012, UAI.

[7]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[8]  David C. Parkes,et al.  A Robust Bayesian Truth Serum for Small Populations , 2012, AAAI.

[9]  Aaron D. Shaw,et al.  Designing incentives for inexpert human raters , 2011, CSCW.

[10]  Craig Boutilier,et al.  Eliciting forecasts from self-interested experts: scoring rules for decision makers , 2011, AAMAS.

[11]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[12]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[13]  Javier R. Movellan,et al.  Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , 2009, NIPS.

[14]  Alex Rogers,et al.  A scoring rule-based mechanism for aggregate demand prediction in the smart grid , 2012, AAMAS.

[15]  R. L. Winkler,et al.  Scoring Rules for Continuous Probability Distributions , 1976 .

[16]  Tim Roughgarden,et al.  Algorithmic Game Theory , 2007 .

[17]  Ruggiero Cavallo,et al.  Efficient crowdsourcing contests , 2012, AAMAS.

[18]  Peng Dai,et al.  Decision-Theoretic Control of Crowd-Sourced Workflows , 2010, AAAI.