Will the Crowd Game the Algorithm?: Using Layperson Judgments to Combat Misinformation on Social Media by Downranking Distrusted Sources

How can social media platforms fight the spread of misinformation? One possibility is to use newsfeed algorithms to downrank content from sources that users rate as untrustworthy. But will laypeople be handicapped by motivated reasoning or lack of expertise, and thus unable to identify misinformation sites? And will they "game" this crowdsourcing mechanism in order to promote content that aligns with their partisan agendas? We conducted a survey experiment in which =984 Americans indicated their trust in numerous news sites. To study the tendency of people to game the system, half of the participants were told their responses would inform social media ranking algorithms. Participants trusted mainstream sources much more than hyper-partisan or fake news sources, and their ratings were highly correlated with professional fact-checker judgments. Critically, informing participants that their responses would influence ranking algorithms did not diminish these results, despite the manipulation increasing the political polarization of trust ratings.

[1]  Jeffrey A. Gottfried,et al.  News use across social media platforms 2016 , 2016 .

[2]  Pankaj K. Agarwal,et al.  Toward Computational Fact-Checking , 2014, Proc. VLDB Endow..

[3]  Mor Naaman,et al.  The Role of Source, Headline and Expressive Responding in Political News Evaluation , 2018 .

[4]  Xuezhi Wang,et al.  Relevant Document Discovery for Fact-Checking Articles , 2018, WWW.

[5]  D. Kahan Misconceptions, Misinformation, and the Logic of Identity-Protective Cognition , 2017 .

[6]  Eunsol Choi,et al.  Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking , 2017, EMNLP.

[7]  Yejin Choi,et al.  Syntactic Stylometry for Deception Detection , 2012, ACL.

[8]  Christo Wilson,et al.  Linguistic Signals under Misinformation and Fact-Checking , 2018, Proc. ACM Hum. Comput. Interact..

[9]  B. Rockenbach,et al.  Measuring lying aversion , 2013 .

[10]  Suhang Wang,et al.  Fake News Detection on Social Media: A Data Mining Perspective , 2017, SKDD.

[11]  Jeffrey T. Hancock,et al.  Fake News in the News: An Analysis of Partisan Coverage of the Fake News Phenomenon , 2018, CSCW Companion.

[12]  Matthew O. Jackson,et al.  Naïve Learning in Social Networks and the Wisdom of Crowds , 2010 .

[13]  Reliance on emotion promotes belief in fake news , 2020, Cognitive research: principles and implications.

[14]  Johan Bollen,et al.  Computational Fact Checking from Knowledge Networks , 2015, PloS one.

[15]  David G. Rand,et al.  The Implied Truth Effect: Attaching Warnings to a Subset of Fake News Headlines Increases Perceived Accuracy of Headlines Without Warnings , 2019, Manag. Sci..

[16]  Philip E. Converse,et al.  ASSESSING THE CAPACITY OF MASS ELECTORATES , 2000 .

[17]  David G. Rand,et al.  Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning , 2019, Cognition.

[18]  Oliver A. McClellan,et al.  Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents , 2019, Research & Politics.

[19]  Tim Weninger,et al.  Fact Checking in Heterogeneous Information Networks , 2016, WWW.

[20]  Bence Bago,et al.  Fake news, fast and slow: Deliberation reduces belief in false (but not true) news headlines. , 2020, Journal of experimental psychology. General.

[21]  Simon Gächter,et al.  Intrinsic Honesty and the Prevalence of Rule Violations across Societies , 2016, Nature.

[22]  Benno Stein,et al.  A Stylometric Inquiry into Hyperpartisan and Fake News , 2017, ACL.

[23]  Michael A. Horning,et al.  FeedReflect: A Tool for Nudging Users to Assess News Credibility on Twitter , 2018, CSCW Companion.

[24]  David G. Rand,et al.  Fighting misinformation on social media using crowdsourced judgments of news source quality , 2018, Proceedings of the National Academy of Sciences.

[25]  Tom Feltwell,et al.  Rethinking Engagement with Online News through Social and Visual Co-Annotation , 2018, CHI.

[26]  Sebastian Tschiatschek,et al.  Fake News Detection in Social Networks via Crowd Signals , 2017, WWW.

[27]  Besnik Fetahu,et al.  Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments , 2019, CHI.

[28]  Miriam J. Metzger,et al.  The science of fake news , 2018, Science.

[29]  Georgios Zervas,et al.  Fake It Till You Make It: Reputation, Competition, and Yelp Review Fraud , 2015, Manag. Sci..

[30]  David G. Rand,et al.  Emphasizing publishers does not effectively reduce susceptibility to misinformation on social media , 2020, Harvard Kennedy School Misinformation Review.

[31]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[32]  Angel Ortega,et al.  Automated Assistants to Identify and Prompt Action on Visual News Bias , 2017, CHI Extended Abstracts.

[33]  Sinan Aral,et al.  The spread of true and false news online , 2018, Science.

[34]  Naeemul Hassan,et al.  The Quest to Automate Fact-Checking , 2015 .

[35]  Bernhard Schölkopf,et al.  Leveraging the Crowd to Detect and Reduce the Spread of Fake News and Misinformation , 2017, WSDM.