Inducing Honest Reporting of Private Information in the Presence of Social Projection

We discuss payment structures that induce honest reporting of private information by risk-neutral agents in settings involving multiple-choice questions. Such payment structures do not rely on the existence of ground-truth answers, but instead they rely on the assumption that agents exhibit social projection. Social projection is a strong form of the well-known psychological phenomenon called the false-consensus effect, where an agent believes that his private answer to a multiple-choice question is the most popular answer. From a theoretical perspective, we first show that when social projection holds true, honest reporting strictly maximizes an agent’s expected reward from a payment structure that simply compares agents’ reported answers and rewards agreements. Furthermore, we suggest how to induce honest reporting by taking the distance between reported answers into account when social projection is strong, i. e., when an agent believes that his private answer is more likely to be reported by a random peer than all the other answers combined. We also discuss how to derive the above results in terms of proper scoring rules. From an empirical perspective, we investigate the consequences of using a payment structure that rewards agreements in a content-analysis experiment on Amazon Mechanical Turk. We obtain some evidence that, under such a payment structure, agents report more accurate answers than when there are no direct incentives for honest reporting of private answers. Moreover, we find that priming agents by briefly mentioning the theoretical properties of the underlying payment structure results in even more accurate answers.

[1]  Boi Faltings,et al.  A Robust Bayesian Truth Serum for Non-Binary Signals , 2013, AAAI.

[2]  Paul Resnick,et al.  Eliciting Informative Feedback: The Peer-Prediction Method , 2005, Manag. Sci..

[3]  David C. Parkes,et al.  Peer prediction without a common prior , 2012, EC '12.

[4]  Kate Larson,et al.  Sharing Rewards Among Strangers Based on Peer Evaluations , 2012, Decis. Anal..

[5]  A. H. Murphy A Note on the Ranked Probability Score , 1971 .

[6]  H. Barkema,et al.  Feelings of Pride and Respect as Drivers of Ongoing Member Activity on Crowdsourcing Platforms , 2015 .

[7]  Rebecca Weiss,et al.  Optimally aggregating elicited expertise : a proposed application of the Bayesian Truth Serum for policy analysis , 2009 .

[8]  Panagiotis G. Ipeirotis Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[9]  T. Griffiths,et al.  Modeling Individual Differences with Dirichlet Processes , 2005 .

[10]  Emmanuel M. Pothos,et al.  Social Projection and a Quantum Approach for Behavior in Prisoner's Dilemma , 2012 .

[11]  A. H. Murphy,et al.  “Good” Probability Assessors , 1968 .

[12]  A. Nederhof Methods of coping with social desirability bias: A review. , 1985 .

[13]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[14]  Drazen Prelec,et al.  Creating Truth-Telling Incentives with the Bayesian Truth Serum , 2013 .

[15]  Jane Taylor Little Ann, and other poems , 2008 .

[16]  Lyle Brenner,et al.  Preference, projection, and packing: Support theory models of judgments of others' preferences , 2011 .

[17]  Tuomas Sandholm,et al.  Decision rules and decision markets , 2010, AAMAS.

[18]  A. Tversky,et al.  Support theory: A nonextensional representation of subjective probability. , 1994 .

[19]  Alexander I. Rudnicky,et al.  Using the Amazon Mechanical Turk for transcription of spoken language , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[20]  Pascal Poupart,et al.  Bayesian Reputation Modeling in E-Marketplaces Sensitive to Subjectivity, Deception and Change , 2006, AAAI.

[21]  Arthur Carvalho Tailored proper scoring rules elicit decision weights , 2015, Judgment and Decision Making.

[22]  D. Prelec A Bayesian Truth Serum for Subjective Data , 2004, Science.

[23]  Gary Marks,et al.  Thinking One's Abilities Are Unique and One's Opinions Are Common , 1984 .

[24]  Laura A. Dabbish,et al.  Designing games with a purpose , 2008, CACM.

[25]  J. Busemeyer,et al.  A quantum probability explanation for violations of ‘rational’ decision theory , 2009, Proceedings of the Royal Society B: Biological Sciences.

[26]  Martin Schader,et al.  Personalized task recommendation in crowdsourcing information systems - Current state of the art , 2014, Decis. Support Syst..

[27]  Jie Ren,et al.  Increasing the crowd's capacity to create: how alternative generation affects the diversity, relevance and effectiveness of generated ads , 2014, Decis. Support Syst..

[28]  J. Sonnemans,et al.  A Truth Serum for Non-Bayesians: Correcting Proper Scoring Rules for Risk Attitudes* , 2009 .

[29]  Y. Yinon,et al.  Age and the false-consensus effect. , 1994, The Journal of social psychology.

[30]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[31]  Yiling Chen,et al.  39 Information Elicitation Sans Verification , 2013 .

[32]  W. Batchelder,et al.  Culture as Consensus: A Theory of Culture and Informant Accuracy , 1986 .

[33]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[34]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[35]  S. Plous The psychology of judgment and decision making , 1994 .

[36]  Brian Mullen,et al.  The false consensus effect: A meta-analysis of 115 hypothesis tests , 1985 .

[37]  T. Griffiths,et al.  Modeling individual differences using Dirichlet processes , 2006 .

[38]  Richard W. Olshavsky,et al.  The False Consensus Effect in Estimates of Smoking Prevalence , 1983 .

[39]  Marilyn A. Walker,et al.  That is your evidence?: Classifying stance in online political debate , 2012, Decis. Support Syst..

[40]  Antonio Torralba,et al.  Describing Visual Scenes using Transformed Dirichlet Processes , 2005, NIPS.

[41]  David C. Parkes,et al.  A Robust Bayesian Truth Serum for Small Populations , 2012, AAAI.

[42]  Aaron D. Shaw,et al.  Designing incentives for inexpert human raters , 2011, CSCW.

[43]  Efraim Turban,et al.  What can crowdsourcing do for decision support? , 2014, Decis. Support Syst..

[44]  Wai-Tat Fu,et al.  Enhancing reliability using peer consistency evaluation in human computation , 2013, CSCW '13.

[45]  M. Cremer,et al.  Creating Truthtelling Incentives with the Bayesian Truth Serum , 2009 .

[46]  Edward I. George,et al.  A bayesian model for collaborative filtering , 1999, AISTATS.

[47]  Zoubin Ghahramani,et al.  Unsupervised and Constrained Dirichlet Process Mixture Models for Verb Clustering , 2009 .

[48]  Glenn Geher,et al.  We think you agree: The detrimental impact of the false consensus effect on behavior , 2002 .

[49]  L. Ross,et al.  The “false consensus effect”: An egocentric bias in social perception and attribution processes , 1977 .

[50]  Robin Hanson,et al.  Combinatorial Information Market Design , 2003, Inf. Syst. Frontiers.

[51]  R. Dawes Statistical criteria for establishing a truly false consensus effect , 1989 .

[52]  William B. Hansen,et al.  Social Projection and Social Conformity in Adolescent Alcohol Use: A Longitudinal Analysis , 1992 .

[53]  F. Allport,et al.  Students' attitudes; a report of the Syracuse University reaction study. , 1931 .

[54]  J. Krueger,et al.  Social Projection to Ingroups and Outgroups: A Review and Meta-Analysis , 2005, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[55]  Ian A. Kash,et al.  Information elicitation for decision making , 2011, AAMAS.

[56]  Kate Larson,et al.  The output-agreement method induces honest behavior in the presence of social projection , 2014, SECO.

[57]  K. Arrow Essays in the theory of risk-bearing , 1958 .

[58]  S. Holstein,et al.  A Family of Strictly Proper Scoring Rules Which Are Sensitive to Distance , 1970 .

[59]  Kate Larson,et al.  A Study on the Influence of the Number of MTurkers on the Quality of the Aggregate Output , 2014, EUMAS.

[60]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[61]  J. D. Campbell Similarity and uniqueness: the effects of attribute type, relevance, and individual differences in self-esteem and depression. , 1986, Journal of personality and social psychology.

[62]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[63]  Efraim Turban,et al.  Introduction to the Special Issue on "Crowdsourcing and Social Networks Analysis" , 2014, Decision Support Systems.

[64]  Edward S. Epstein,et al.  A Scoring System for Probability Forecasts of Ranked Categories , 1969 .

[65]  Craig Boutilier,et al.  Eliciting forecasts from self-interested experts: scoring rules for decision makers , 2011, AAMAS.

[66]  Estevam R. Hruschka,et al.  Tweet sentiment analysis with classifier ensembles , 2014, Decis. Support Syst..

[67]  Stanko Dimitrov,et al.  Composition of markets with conflicting incentives , 2010, EC '10.

[68]  Robert L. Winkler,et al.  Sensitivity to Distance and Baseline Distributions in Forecast Evaluation , 2009, Manag. Sci..

[69]  Norman Miller,et al.  Ten years of research on the false-consensus effect: an empirical and theoretical review , 1987 .

[70]  Manu Sridharan,et al.  Predicting your own effort , 2012, AAMAS.

[71]  Aaron D. Shaw,et al.  Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India , 2012, CHI.

[72]  Saif Mohammad,et al.  From once upon a time to happily ever after: Tracking emotions in mail and books , 2012, Decis. Support Syst..