An Algorithmic Equity Toolkit for Technology Audits by Community Advocates and Activists

A wave of recent scholarship documenting the discriminatory harms of algorithmic systems has spurred widespread interest in algorithmic accountability and regulation. Yet effective accountability and regulation is stymied by a persistent lack of resources supporting public understanding of algorithms and artificial intelligence. Through interactions with a US-based civil rights organization and their coalition of community organizations, we identify a need for (i) heuristics that aid stakeholders in distinguishing between types of analytic and information systems in lay language, and (ii) risk assessment tools for such systems that begin by making algorithms more legible. The present work delivers a toolkit to achieve these aims. This paper both presents the Algorithmic Equity Toolkit (AEKit) Equity as an artifact, and details how our participatory process shaped its design. Our work fits within human-computer interaction scholarship as a demonstration of the value of HCI methods and approaches to problems in the area of algorithmic transparency and accountability.

[1]  Sophie Bishop,et al.  Managing visibility on YouTube through algorithmic gossip , 2019, New Media Soc..

[2]  Tim Miller,et al.  Explanation in Artificial Intelligence: Insights from the Social Sciences , 2017, Artif. Intell..

[3]  Julia Powles,et al.  "Meaningful Information" and the Right to Explanation , 2017, FAT.

[4]  Alex Rosenblat,et al.  Algorithmic Labor and Information Asymmetries: A Case Study of Uber’s Drivers , 2016 .

[5]  William W. Gaver.  What should we expect from research through design? , 2012, CHI.

[6]  Irina Shklovski,et al.  The Algorithm and the User: How Can HCI Use Lay Understandings of Algorithmic Systems? , 2018, CHI Extended Abstracts.

[7]  Danah Boyd,et al.  Fairness and Abstraction in Sociotechnical Systems , 2019, FAT.

[8]  Jeremy P. Birnholtz,et al.  How People Form Folk Theories of Social Media Feeds and What it Means for How We Study Self-Presentation , 2018, CHI.

[9]  Phoebe Sengers,et al.  Reflections on Design Methods for Underserved Communities , 2017, CSCW Companion.

[10]  Hany Farid,et al.  The accuracy, fairness, and limits of predicting recidivism , 2018, Science Advances.

[11]  Munmun De Choudhury,et al.  Where is the Human?: Bridging the Gap Between AI and HCI , 2019, CHI Extended Abstracts.

[12]  Ben Green,et al.  Data Science as Political Action: Grounding Data Science in a Politics of Justice , 2018, J. Soc. Comput..

[13]  Tawanna Dillahunt,et al.  An intersectional approach to designing in the margins , 2018, Interactions.

[14]  Emily Bienvenue.  Computational propaganda: political parties, politicians, and political manipulation on social media , 2020, International Affairs.

[15]  Karrie Karahalios,et al.  Communicating Algorithmic Process in Online Behavioral Advertising , 2018, CHI.

[16]  Joan Donovan,et al.  Source hacking: media manipulation in practice , 2019 .

[17]  S. L. Star,et al.  The Ethnography of Infrastructure , 1999 .

[19]  N. Allum,et al.  Science in Society: Re-Evaluating the Deficit Model of Public Attitudes , 2004 .

[20]  Dear Mr Sotiropoulos.  ARTICLE 29 Data Protection Working Party , 2013 .

[21]  John Zimmerman,et al.  Research through design as a method for interaction design research in HCI , 2007, CHI.

[22]  Batya Friedman,et al.  Toward inclusive tech policy design: a method for underrepresented voices to strengthen tech policy documents , 2019, Ethics and Information Technology.

[23]  Chris Russell,et al.  Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR , 2017, ArXiv.

[24]  Joshua A. Tucker,et al.  Social Media, Political Polarization, and Political Disinformation: A Review of the Scientific Literature , 2018 .

[25]  Jenna Burrell,et al.  How the machine ‘thinks’: Understanding opacity in machine learning algorithms , 2016 .

[26]  Rebecca Gray,et al.  Understanding User Beliefs About Algorithmic Curation in the Facebook News Feed , 2015, CHI.

[27]  Anselm L. Strauss,et al.  The Articulation of Work Through Interaction , 1993 .

[28]  Eric Gilbert,et al.  User Attitudes towards Algorithmic Opacity and Transparency in Online Reviewing Platforms , 2019, CHI.

[29]  Taina Bucher,et al.  Want to be on the top? Algorithmic power and the threat of invisibility on Facebook , 2012, New Media Soc..

[30]  Michael Veale,et al.  Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For , 2017 .

[31]  Emilee J. Rader,et al.  Explanations as Mechanisms for Supporting Algorithmic Transparency , 2018, CHI.

[32]  Neil Gandal,et al.  Price Manipulation in the Bitcoin Ecosystem , 2017 .

[33]  Kim Halskov,et al.  Documenting the Research Through Design Process , 2016, Conference on Designing Interactive Systems.

[34]  Bryce Clayton Newell,et al.  A typology of privacy , 2016 .

[35]  Gina Neff,et al.  Imagined Affordance: Reconstructing a Keyword for Communication Theory , 2015 .

[36]  Alex 'Sandy' Pentland,et al.  An Experimental Study of Cryptocurrency Market Dynamics , 2018, CHI.

[37]  Timnit Gebru,et al.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification , 2018, FAT.

[38]  A. Hoffmann.  Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse , 2019, Information, Communication & Society.

[39]  Shaowen Bardzell,et al.  Immodest Proposals: Research Through Design and Knowledge , 2015, CHI.

[40]  Allison Woodruff,et al.  A Qualitative Exploration of Perceptions of Algorithmic Fairness , 2018, CHI.

[41]  Kelley Cotter,et al.  Playing the visibility game: How digital influencers and algorithms negotiate influence on Instagram , 2018, New Media Soc..