Partial Truthfulness in Minimal Peer Prediction Mechanisms with Limited Knowledge

We study minimal single-task peer prediction mechanisms that have limited knowledge about agents' beliefs. Without knowing what agents' beliefs are or eliciting additional information, it is not possible to design a truthful mechanism in a Bayesian-Nash sense. We go beyond truthfulness and explore equilibrium strategy profiles that are only partially truthful. Using the results from the multi-armed bandit literature, we give a characterization of how inefficient these equilibria are comparing to truthful reporting. We measure the inefficiency of such strategies by counting the number of dishonest reports that any minimal knowledge-bounded mechanism must have. We show that the order of this number is $\Theta(\log n)$, where $n$ is the number of agents, and we provide a peer prediction mechanism that achieves this bound in expectation.

[1]  Boi Faltings,et al.  A Robust Bayesian Truth Serum for Non-Binary Signals , 2013, AAAI.

[2]  Paul Resnick,et al.  Eliciting Informative Feedback: The Peer-Prediction Method , 2005, Manag. Sci..

[3]  Boi Faltings,et al.  Incentives to Counter Bias in Human Computation , 2014, HCOMP.

[4]  Boi Faltings,et al.  Incentives for Effort in Crowdsourcing Using the Peer Truth Serum , 2016, ACM Trans. Intell. Syst. Technol..

[5]  Aurélien Garivier,et al.  The KL-UCB Algorithm for Bounded Stochastic Bandits and Beyond , 2011, COLT.

[6]  Boi Faltings,et al.  Game Theory for Data Science: Eliciting Truthful Information , 2017, Game Theory for Data Science.

[7]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[8]  D. Prelec.  A Bayesian Truth Serum for Subjective Data , 2004, Science.

[9]  Yiling Chen,et al.  Output Agreement Mechanisms and Common Knowledge , 2014, HCOMP.

[10]  Apostolos Burnetas,et al.  Optimal Adaptive Policies for Markov Decision Processes , 1997, Math. Oper. Res..

[11]  Jens Witkowski.  Robust peer prediction mechanisms , 2015 .

[12]  Boi Faltings,et al.  Peer Truth Serum: Incentives for Crowdsourcing Measurements and Opinions , 2017, ArXiv.

[13]  Boi Faltings,et al.  Incentives for Answering Hypothetical Questions , 2011 .

[14]  Boi Faltings,et al.  Incentives for Truthful Information Elicitation of Continuous Signals , 2014, AAAI.

[15]  Boi Faltings,et al.  Incentives for expressing opinions in online polls , 2008, EC '08.

[16]  T. L. Lai Andherbertrobbins.  Asymptotically Efficient Adaptive Allocation Rules , 2022 .

[17]  V. Shnayder,et al.  Informed Truthfulness in MultiTask Peer Prediction ( Working paper ) , 2016 .

[18]  R. Agrawal.  Sample mean based index policies by O(log n) regret for the multi-armed bandit problem , 1995, Advances in Applied Probability.

[19]  Csaba Szepesvári,et al.  Exploration-exploitation tradeoff using variance estimates in multi-armed bandits , 2009, Theor. Comput. Sci..

[20]  Yang Liu,et al.  Learning to Incentivize: Eliciting Effort via Output Agreement , 2016, IJCAI.

[21]  Anirban Dasgupta,et al.  Crowdsourced judgement elicitation with endogenous proficiency , 2013, WWW.

[22]  Jens Witkowski,et al.  A Geometric Method to Construct Minimal Peer Prediction Mechanisms , 2016, AAAI.

[23]  Kevin Leyton-Brown,et al.  Incentivizing Evaluation via Limited Access to Ground Truth: Peer-Prediction Makes Things Worse , 2016, ArXiv.

[24]  David C. Parkes,et al.  A Robust Bayesian Truth Serum for Small Populations , 2012, AAAI.

[25]  Arpit Agarwal,et al.  Informed Truthfulness in Multi-Task Peer Prediction , 2016, EC.

[26]  Jean-Yves Audibert,et al.  Regret Bounds and Minimax Policies under Partial Monitoring , 2010, J. Mach. Learn. Res..

[27]  Grant Schoenebeck,et al.  Equilibrium Selection in Information Elicitation without Verification via Information Monotonicity , 2018, ITCS.

[28]  David C. Parkes,et al.  Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics , 2016, IJCAI.

[29]  Boi Faltings,et al.  Swissnoise: Online Polls with Game-Theoretic Incentives , 2014, AAAI.

[30]  David C. Parkes,et al.  Peer prediction without a common prior , 2012, EC '12.

[31]  Kannan Ramchandran,et al.  Truth Serums for Massively Crowdsourced Evaluation Tasks , 2015, ArXiv.