Incentives for Subjective Evaluations with Private Beliefs

The modern web critically depends on aggregation of information from self-interested agents, for example opinion polls, product ratings, or crowdsourcing. We consider a setting where multiple objects (questions, products, tasks) are evaluated by a group of agents. We first construct a minimal peer prediction mechanism that elicits honest evaluations from a homogeneous population of agents with different private beliefs. Second, we show that it is impossible to strictly elicit honest evaluations from a heterogeneous group of agents with different private beliefs. Nevertheless, we provide a modified version of a divergence-based Bayesian Truth Serum that incentivizes agents to report consistently, making truthful reporting a weak equilibrium of the mechanism.

[1]  Yoav Shoham,et al.  Truthful Surveys , 2008, WINE.

[2]  Yiling Chen,et al.  Output Agreement Mechanisms and Common Knowledge , 2014, HCOMP.

[3]  David C. Parkes,et al.  Peer prediction without a common prior , 2012, EC '12.

[4]  Blake Riley,et al.  Minimum Truth Serums with Optional Predictions , 2014 .

[5]  Anirban Dasgupta,et al.  Crowdsourced judgement elicitation with endogenous proficiency , 2013, WWW.

[6]  Yoav Shoham,et al.  Eliciting truthful answers to multiple-choice questions , 2009, EC '09.

[7]  Boi Faltings,et al.  Collusion-resistant, incentive-compatible feedback payments , 2007, EC '07.

[8]  Robin Hanson,et al.  Combinatorial Information Market Design , 2003, Inf. Syst. Frontiers.

[9]  David C. Parkes,et al.  A Robust Bayesian Truth Serum for Small Populations , 2012, AAAI.

[10]  David C. Parkes,et al.  Learning the Prior in Minimal Peer Prediction , 2013 .

[11]  D. Prelec.  A Bayesian Truth Serum for Subjective Data , 2004, Science.

[12]  David M. Pennock,et al.  A Utility Framework for Bounded-Loss Market Makers , 2007, UAI.

[13]  David M. Pennock,et al.  Collective revelation: a mechanism for self-verified, weighted, and truthful predictions , 2009, EC '09.

[14]  Yiling Chen,et al.  Elicitability and knowledge-free elicitation with peer prediction , 2014, AAMAS.

[15]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[16]  Boi Faltings,et al.  Minimum payments that reward honest reputation feedback , 2006, EC '06.

[17]  Boi Faltings,et al.  Incentives for Answering Hypothetical Questions , 2011 .

[18]  Boi Faltings,et al.  Incentives for Truthful Information Elicitation of Continuous Signals , 2014, AAAI.

[19]  Yiling Chen,et al.  39 Information Elicitation Sans Verification , 2013 .

[20]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[21]  Boi Faltings,et al.  A Robust Bayesian Truth Serum for Non-Binary Signals , 2013, AAAI.

[22]  Paul Resnick,et al.  Eliciting Informative Feedback: The Peer-Prediction Method , 2005, Manag. Sci..