Peer Prediction with Heterogeneous Users

Peer prediction mechanisms incentivize agents to truthfully report their signals, in the absence of a verification mechanism, by comparing their reports with those of their peers. Prior work in this area is essentially restricted to the case of homogeneous agents, whose signal distributions are identical. This is limiting in many domains, where we would expect agents to differ in taste, judgment, and reliability. Although the Correlated Agreement (CA) mechanism [Shnayder et al. 2016a] can be extended to handle heterogeneous agents, there is a new challenge of efficiently estimating agent signal types. We solve this problem by clustering agents based on their reporting behavior, proposing a mechanism that works with clusters of agents, and designing algorithms that learn such a clustering. In this way, we also connect peer prediction with the Dawid and Skene [1979] literature on latent types. We retain the robustness against coordinated misreports of the CA mechanism, achieving an approximate incentive guarantee of ε-informed truthfulness. We show on real data that this incentive approximation is reasonable in practice, even with a small number of clusters.

[1]  Eric Horvitz,et al.  Geographic and Temporal Trends in Fake News Consumption During the 2016 US Presidential Election , 2017, CIKM.

[2]  Jinwoo Shin,et al.  Optimality of Belief Propagation for Crowdsourced Classification , 2016, ICML.

[3]  Devavrat Shah,et al.  Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems , 2011, Oper. Res..

[4]  Yang Liu,et al.  Sequential Peer Prediction: Learning to Elicit Effort using Posted Prices , 2016, AAAI.

[5]  Anirban Dasgupta,et al.  Crowdsourced judgement elicitation with endogenous proficiency , 2013, WWW.

[6]  D. Prelec.  A Bayesian Truth Serum for Subjective Data , 2004, Science.

[7]  Boi Faltings,et al.  Incentives for Subjective Evaluations with Private Beliefs , 2015, AAAI.

[8]  Stephen J. Roberts,et al.  Dynamic Bayesian Combination of Multiple Imperfect Classifiers , 2012, Decision Making and Imperfection.

[9]  Nihar B. Shah,et al.  Regularized Minimax Conditional Entropy for Crowdsourcing , 2015, ArXiv.

[10]  Xi Chen,et al.  Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing , 2014, J. Mach. Learn. Res..

[11]  Boi Faltings,et al.  Incentive Schemes for Participatory Sensing , 2015, AAMAS.

[12]  Justin Cheng,et al.  Peer and self assessment in massive online classes , 2013, ACM Trans. Comput. Hum. Interact..

[13]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[14]  Grant Schoenebeck,et al.  Putting Peer Prediction Under the Micro(economic)scope and Making Truth-Telling Focal , 2016, WINE.

[15]  Ashish Khetan,et al.  Achieving budget-optimality with adaptive schemes in crowdsourcing , 2016, NIPS.

[16]  Boi Faltings,et al.  Incentives for Effort in Crowdsourcing Using the Peer Truth Serum , 2016, ACM Trans. Intell. Syst. Technol..

[17]  Arpit Agarwal,et al.  Informed Truthfulness in Multi-Task Peer Prediction , 2016, EC.

[18]  Purnamrita Sarkar,et al.  Active Learning for Crowd-Sourced Databases , 2012, ArXiv.

[19]  A. P. Dawid,et al.  Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm , 1979 .

[20]  David C. Parkes,et al.  Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics , 2016, IJCAI.

[21]  David C. Parkes,et al.  Practical Peer Prediction for Peer Assessment , 2016, HCOMP.

[22]  Matthew Lease,et al.  SQUARE: A Benchmark for Research on Computing Crowd Consensus , 2013, HCOMP.

[23]  Daniel M. Russell,et al.  Student skill and goal achievement in the mapping with google MOOC , 2014, L@S.

[24]  Boi Faltings,et al.  Mechanisms for Making Crowds Truthful , 2014, J. Artif. Intell. Res..

[25]  Kevin Leyton-Brown,et al.  Incentivizing Evaluation via Limited Access to Ground Truth: Peer-Prediction Makes Things Worse , 2016, ArXiv.

[26]  Anima Anandkumar,et al.  Tensor decompositions for learning latent variable models , 2012, J. Mach. Learn. Res..

[27]  Krishna P. Gummadi,et al.  Message Impartiality in Social Media Discussions , 2016, ICWSM.

[28]  Fei-Fei Li,et al.  Novel Dataset for Fine-Grained Image Categorization : Stanford Dogs , 2012 .

[29]  Luc Devroye,et al.  Combinatorial methods in density estimation , 2001, Springer series in statistics.

[30]  David C. Parkes,et al.  Learning the Prior in Minimal Peer Prediction , 2013 .

[31]  Grant Schoenebeck,et al.  G T ] 3 M ay 2 01 6 A Framework For Designing Information Elicitation Mechanisms That Reward Truth-telling Yuqing Kong University of Michigan , 2016 .

[32]  Yang Cai,et al.  Optimum Statistical Estimation with Strategic Data Sources , 2014, COLT.

[33]  Kannan Ramchandran,et al.  Truth Serums for Massively Crowdsourced Evaluation Tasks , 2015, ArXiv.

[34]  Debmalya Mandal,et al.  Peer Prediction with Heterogeneous Tasks , 2016, ArXiv.

[35]  Paul Resnick,et al.  Eliciting Informative Feedback: The Peer-Prediction Method , 2005, Manag. Sci..

[36]  Purnamrita Sarkar,et al.  Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning , 2014, Proc. VLDB Endow..

[37]  Boi Faltings,et al.  Game Theory for Data Science: Eliciting Truthful Information , 2017, Game Theory for Data Science.

[38]  David C. Parkes,et al.  Dwelling on the Negative: Incentivizing Effort in Peer Prediction , 2013, HCOMP.

[39]  M. Gentzkow,et al.  Social Media and Fake News in the 2016 Election , 2017 .

[40]  Jens Witkowski,et al.  A Geometric Perspective on Minimal Peer Prediction , 2017, ACM Trans. Economics and Comput..

[41]  Yang Liu,et al.  Machine-Learning Aided Peer Prediction , 2017, EC.