Auditing the Partisanship of Google Search Snippets

The text snippets presented in web search results provide users with a slice of page content that they can quickly scan to help inform their click decisions. However, little is known about how these snippets are generated or how they relate to a user's search query. Motivated by the growing body of evidence suggesting that search engine rankings can influence undecided voters, we conducted an algorithm audit of the political partisanship of Google Search snippets relative to the webpages they are extracted from. To accomplish this, we constructed lexicon of partisan cues to measure partisanship and construct a set of left- and right-leaning search queries. Then, we collected a large dataset of Search Engine Results Pages (SERPs) by running our partisan queries and their autocomplete suggestions on Google Search. After using our lexicon to score the machine-coded partisanship of snippets and webpages, we found that Google Search's snippets generally amplify partisanship, and that this effect is robust across different types of webpages, query topics, and partisan (left- and right-leaning) queries.

[1]  Balachander Krishnamurthy,et al.  Measuring personalization of web search , 2013, WWW.

[2]  Tim Weninger,et al.  Consumers and Curators: Browsing and Voting Patterns on Reddit , 2017, IEEE Transactions on Computational Social Systems.

[3]  Jan Kleinnijenhuis,et al.  Measurement of party positions on the basis of party programmes, media coverage and voter perceptions , 2001 .

[4]  H. P. Edmundson,et al.  New Methods in Automatic Extracting , 1969, JACM.

[5]  Stefan Kaufmann,et al.  Language and Ideology in Congress , 2011, British Journal of Political Science.

[6]  Hugo Larochelle,et al.  EACL 2014 14th Conference of the European Chapter of the Association for Computational Linguistics Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC) , 2014 .

[7]  Marti A. Hearst,et al.  Improving Search Results Quality by Customizing Summary Lengths , 2008, ACL.

[8]  N. Diakopoulos.  Algorithmic Accountability Reporting: On the Investigation of Black Boxes , 2014 .

[9]  Kam-Fai Wong,et al.  Extractive Summarization Using Supervised and Semi-Supervised Learning , 2008, COLING.

[10]  Hans Peter Luhn,et al.  The Automatic Creation of Literature Abstracts , 1958, IBM J. Res. Dev..

[11]  Ireland Miranda De Vries,et al.  Estimating policy positions from the computer coding of political texts: results from Italy, the Netherlands and Ireland , 2003 .

[12]  Mirella Lapata,et al.  Neural Summarization by Extracting Sentences and Words , 2016, ACL.

[13]  Ronald E. Robertson,et al.  The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections , 2015, Proceedings of the National Academy of Sciences.

[14]  David Lazer,et al.  Auditing Partisan Audience Bias within Google Search , 2018, Proc. ACM Hum. Comput. Interact..

[15]  David Lazer,et al.  A Frame of Mind: Using Statistical Models for Detection of Framing and Agenda Setting Campaigns , 2015, ACL.

[16]  Matthias Hagen,et al.  A User Study on Snippet Generation: Text Reuse vs. Paraphrases , 2018, SIGIR.

[17]  Philip J. Stone,et al.  Extracting Information. (Book Reviews: The General Inquirer. A Computer Approach to Content Analysis) , 1967 .

[18]  M. Laver,et al.  Estimating policy positions from political texts , 2000 .

[19]  Marshall S. Smith,et al.  The general inquirer: A computer approach to content analysis. , 1967 .

[20]  G. Lakoff.  Moral Politics: How Liberals and Conservatives Think , 1996 .

[21]  Richard I. Hofferbert,et al.  Parties, Policies, And Democracy , 1994 .

[22]  J. Vandello,et al.  Prevalence of Rape Myths in Headlines and Their Effects on Attitudes Toward Rape , 2008 .

[23]  Karrie Karahalios,et al.  Auditing Algorithms : Research Methods for Detecting Discrimination on Internet Platforms , 2014 .

[24]  Bowen Zhou,et al.  SummaRuNNer: A Recurrent Neural Network Based Sequence Model for Extractive Summarization of Documents , 2016, AAAI.

[25]  Krishna P. Gummadi,et al.  Media Bias Monitor: Quantifying Biases of Social Media News Outlets at Large-Scale , 2018, ICWSM.

[26]  David Lazer,et al.  Location, Location, Location: The Impact of Geolocation on Web Search Personalization , 2015, Internet Measurement Conference.

[27]  I. Budge,et al.  Mapping Policy Preferences: Estimates for Parties, Electors, and Governments 1945-1998 , 2001 .

[28]  Philip J. Stone,et al.  The general inquirer: A computer system for content analysis and retrieval based on the sentence as a unit of information , 2007 .

[29]  Philip E. Tetlock,et al.  Verbal Style and the Presidency: A Computer-Based Analysis , 1985 .

[30]  J. Charteris-Black.  Politicians and Rhetoric: The Persuasive Power of Metaphor , 2004 .

[31]  Wei-Hao Lin,et al.  A Joint Topic and Perspective Model for Ideological Discourse , 2008, ECML/PKDD.

[32]  Thorsten Joachims,et al.  Eye-tracking analysis of user behavior in WWW search , 2004, SIGIR '04.

[33]  Lada A. Adamic,et al.  Exposure to ideologically diverse news and opinion on Facebook , 2015, Science.

[34]  I. Budge,et al.  Ideology, strategy and party change : spatial analyses of post-war election programmes in 19 democracies , 1987 .

[35]  David Lazer,et al.  Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages , 2018, WWW.

[36]  Edward Cutrell,et al.  What are you looking for?: an eye-tracking study of information usage in web search , 2007, CHI.

[37]  Jeffrey A. Gottfried,et al.  News use across social media platforms 2016 , 2016 .

[38]  Chris D. Paice,et al.  Constructing literature abstracts by computer: Techniques and prospects , 1990, Inf. Process. Manag..

[39]  D. Tambini,et al.  Digital Dominance: The Power of Google, Amazon, Facebook and Apple , 2018 .

[40]  P. Stone,et al.  Verbal Style and the Presidency: A Computer-Based Analysis. , 1985 .

[41]  Devdatt P. Dubhashi,et al.  Extractive Summarization using Continuous Vector Space Models , 2014, CVSC@EACL.

[42]  Mark Sanderson,et al.  Advantages of query biased summaries in information retrieval , 1998, SIGIR '98.

[43]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[44]  Noah A. Smith,et al.  Measuring Ideological Proportions in Political Speeches , 2013, EMNLP.

[45]  Justin M. Rao,et al.  Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis , 2016 .

[46]  M. Laver,et al.  Extracting Policy Positions from Political Texts Using Words as Data , 2003, American Political Science Review.

[47]  Ullrich K. H. Ecker,et al.  The effects of subtle misinformation in news headlines. , 2014, Journal of experimental psychology. Applied.

[48]  Bianca C. Reisdorf,et al.  Search and Politics: The Uses and Impacts of Search in Britain, France, Germany, Italy, Poland, Spain, and the United States , 2017 .

[49]  Hugh E. Williams,et al.  Fast generation of result snippets in web search , 2007, SIGIR.

[50]  F. Tripodi.  Searching for Alternative Facts , 2018 .

[51]  Tamás D. Gedeon,et al.  What Snippet Size is Needed in Mobile Web Search? , 2017, CHIIR.

[52]  Ryen W. White,et al.  Personalizing web search results by reading level , 2011, CIKM '11.

[53]  Tomas Mikolov,et al.  Advances in Pre-Training Distributed Word Representations , 2017, LREC.

[54]  Roderick P. Hart,et al.  Political Tone: How Leaders Talk and Why , 2013 .

[55]  Falk Scholer,et al.  Constructing query-biased summaries: a comparison of human and system generated snippets , 2010, IIiX.

[56]  David Lazer,et al.  Suppressing the Search Engine Manipulation Effect (SEME) , 2017, Proc. ACM Hum. Comput. Interact..

[57]  N. Newman,et al.  Reuters Institute Digital News Report 2019 , 2019 .