Mitigating Bias in Algorithmic Systems: A Fish-Eye View of Problems and Solutions Across Domains

Mitigating bias in algorithmic systems is a critical issue drawing attention across communities within the information and computer sciences. Given the complexity of the problem and the involvement of multiple stakeholders – including developers, end-users and third-parties – there is a need to understand the landscape of the sources of bias, and the solutions being proposed to address them. This survey provides a “fish-eye view," examining approaches across four areas of research. The literature describes three steps toward a comprehensive treatment – bias detection, fairness management and explainability management – and underscores the need to work from within the system as well as from the perspective of stakeholders in the broader context.

[1]  E. Yom-Tov Demographic differences in search engine use with implications for cohort selection , 2018, Information Retrieval Journal.

[2]  Yoshikazu Ikeda,et al.  Explanatory rule extraction based on the trained neural network and the genetic programming , 2006 .

[3]  Carsten Eickhoff,et al.  Cognitive Biases in Crowdsourcing , 2018, WSDM.

[4]  Rayid Ghani,et al.  Aequitas: A Bias and Fairness Audit Toolkit , 2018, ArXiv.

[5]  Carlos Guestrin,et al.  Anchors: High-Precision Model-Agnostic Explanations , 2018, AAAI.

[6]  Lucy Vasserman,et al.  Measuring and Mitigating Unintended Bias in Text Classification , 2018, AIES.

[7]  Stefan Haufe,et al.  On the interpretation of weight vectors of linear models in multivariate neuroimaging , 2014, NeuroImage.

[8]  Christo Wilson,et al.  Investigating the Impact of Gender on Rank in Resume Search Engines , 2018, CHI.

[9]  Bart Baesens,et al.  Performance of classification models from a user perspective , 2011, Decis. Support Syst..

[10]  Desheng Hu,et al.  Auditing the Partisanship of Google Search Snippets , 2019, WWW.

[11]  Abbe Mowshowitz,et al.  Measuring search engine bias , 2005, Inf. Process. Manag..

[12]  Olfa Nasraoui,et al.  Transparency in Fair Machine Learning: the Case of Explainable Recommender Systems , 2018, Human and Machine Learning.

[13]  Ricardo Baeza-Yates,et al.  FA*IR: A Fair Top-k Ranking Algorithm , 2017, CIKM.

[14]  Alex Rosenblat,et al.  Algorithmic Labor and Information Asymmetries: A Case Study of Uber’s Drivers , 2016 .

[15]  Darren Gergle,et al.  The Gendered Geography of Contributions to OpenStreetMap: Complexities in Self-Focus Bias , 2019, CHI.

[16]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[17]  Erik Duval,et al.  Visualizing recommendations to support exploration, transparency and controllability , 2013, IUI '13.

[18]  Sibel Adali,et al.  Rating Reliability and Bias in News Articles: Does AI Assistance Help Everyone? , 2019, ICWSM.

[19]  Karrie Karahalios,et al.  "Be Careful; Things Can Be Worse than They Appear": Understanding Biased Algorithms and Users' Behavior Around Them in Rating Platforms , 2017, ICWSM.

[20]  A. Korolova,et al.  Discrimination through Optimization , 2019, Proc. ACM Hum. Comput. Interact..

[21]  Maria Soledad Pera,et al.  All The Cool Kids, How Do They Fit In?: Popularity and Demographic Biases in Recommender Evaluation and Effectiveness , 2018, FAT.

[22]  Bernard J. Jansen,et al.  An examination of searcher's perceptions of nonsponsored and sponsored links during ecommerce Web searching , 2006, J. Assoc. Inf. Sci. Technol..

[23]  Kristian Lum,et al.  An algorithm for removing sensitive information: Application to race-independent recidivism prediction , 2017, The Annals of Applied Statistics.

[24]  Krishna P. Gummadi,et al.  Human Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction , 2018, WWW.

[25]  Yue Yin,et al.  Explainable Recommendation via Multi-Task Learning in Opinionated Text Data , 2018, SIGIR.

[26]  Ben Green,et al.  Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments , 2019, FAT.

[27]  Paul D. Clough,et al.  Competent Men and Warm Women: Gender Stereotypes and Backlash in Image Search Results , 2017, CHI.

[28]  David Maxwell,et al.  The impact of result diversification on search behaviour and performance , 2019, Information Retrieval Journal.

[29]  Zubair Shafiq,et al.  Measuring Political Personalization of Google News Search , 2019, WWW.

[30]  Noah A. Smith,et al.  Deep Weighted Averaging Classifiers , 2018, FAT.

[31]  Virgílio A. F. Almeida,et al.  Stereotypes in Search Engine Results: Understanding The Role of Local and Global Factors , 2016, 1609.05413.

[32]  George W. Furnas,et al.  A fisheye follow-up: further reflections on focus + context , 2006, CHI.

[33]  Salvatore Ruggieri,et al.  A multidisciplinary survey on discrimination analysis , 2013, The Knowledge Engineering Review.

[34]  Allison Woodruff,et al.  A Qualitative Exploration of Perceptions of Algorithmic Fairness , 2018, CHI.

[35]  Ellen M. Voorhees,et al.  Bias and the limits of pooling for large collections , 2007, Information Retrieval.

[36]  Karrie Karahalios,et al.  Auditing Algorithms : Research Methods for Detecting Discrimination on Internet Platforms , 2014 .

[37]  Jun Sakuma,et al.  Fairness-Aware Classifier with Prejudice Remover Regularizer , 2012, ECML/PKDD.

[38]  Andreas Rauber,et al.  On the relationship between query characteristics and IR functions retrieval bias , 2011, J. Assoc. Inf. Sci. Technol..

[39]  Gerhard Friedrich,et al.  A Taxonomy for Generating Explanations in Recommender Systems , 2011, AI Mag..

[40]  Indre Zliobaite,et al.  Using sensitive personal data may be necessary for avoiding discrimination in data-driven decision models , 2016, Artificial Intelligence and Law.

[41]  Leif Azzopardi,et al.  Algorithmic Bias: Do Good Systems Make Relevant Documents More Retrievable? , 2017, CIKM.

[42]  Carlos Eduardo Scheidegger,et al.  Certifying and Removing Disparate Impact , 2014, KDD.

[43]  Ryen W. White,et al.  Belief Dynamics and Biases in Web Search , 2015, ACM Trans. Inf. Syst..

[44]  Andreas Krause,et al.  Mathematical Notions vs. Human Perception of Fairness: A Descriptive Approach to Fairness for Machine Learning , 2019, KDD.

[45]  Min Kyung Lee Algorithmic Mediation in Group Decisions: Fairness Perceptions of Algorithmically Mediated vs. Discussion-Based Social Division , 2017, CSCW.

[46]  Elias Bareinboim,et al.  Fairness in Decision-Making - The Causal Explanation Formula , 2018, AAAI.

[47]  David García,et al.  Bias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr , 2017, CSCW.

[48]  Franco Turini,et al.  Measuring Discrimination in Socially-Sensitive Decision Records , 2009, SDM.

[49]  Rachel K. E. Bellamy,et al.  AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias , 2018, ArXiv.

[50]  Putra Manggala,et al.  Using Image Fairness Representations in Diversity-Based Re-ranking for Recommendations , 2018, UMAP.

[51]  Erik Strumbelj,et al.  An Efficient Explanation of Individual Classifications using Game Theory , 2010, J. Mach. Learn. Res..

[52]  Lise Getoor,et al.  Personalized explanations for hybrid recommender systems , 2019, IUI.

[53]  Ryan Turner,et al.  A model explanation system , 2016, 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP).

[54]  David Lazer,et al.  Suppressing the Search Engine Manipulation Effect (SEME) , 2017, Proc. ACM Hum. Comput. Interact..

[55]  Lars Niklasson,et al.  Evolving decision trees using oracle guides , 2009, 2009 IEEE Symposium on Computational Intelligence and Data Mining.

[56]  Leif Azzopardi,et al.  Best and Fairest: An Empirical Analysis of Retrieval System Bias , 2014, ECIR.

[57]  Vasant Honavar,et al.  Fairness in Algorithmic Decision Making: An Excursion Through the Lens of Causality , 2019, WWW.

[58]  Yair Zick,et al.  Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[59]  Jieyu Zhao,et al.  Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods , 2018, NAACL.

[60]  Helen Nissenbaum,et al.  Bias in computer systems , 1996, TOIS.

[61]  Leif Azzopardi,et al.  A Retrievability Analysis: Exploring the Relationship Between Retrieval Bias and Retrieval Performance , 2014, CIKM.

[62]  Max Welling,et al.  Visualizing Deep Neural Network Decisions: Prediction Difference Analysis , 2017, ICLR.

[63]  Emre Kıcıman,et al.  Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries , 2018, Front. Big Data.

[64]  Eric Gilbert,et al.  User Attitudes towards Algorithmic Opacity and Transparency in Online Reviewing Platforms , 2019, CHI.

[65]  Matt J. Kusner,et al.  Counterfactual Fairness , 2017, NIPS.

[66]  Latanya Sweeney,et al.  Discrimination in online ad delivery , 2013, CACM.

[67]  Johannes Schöning,et al.  The Effect of Population and "Structural" Biases on Social Media-based Algorithms: A Case Study in Geolocation Inference Across the Urban-Rural Spectrum , 2017, CHI.

[68]  Zhi-Hua Zhou,et al.  Extracting symbolic rules from trained neural network ensembles , 2003, AI Commun..

[69]  Franco Turini,et al.  k-NN as an implementation of situation testing for discrimination discovery and prevention , 2011, KDD.

[70]  Mark Levene,et al.  Presentation bias is significant in determining user preference for search results - A user study , 2009, J. Assoc. Inf. Sci. Technol..

[71]  Gary Klein,et al.  Explaining Explanation, Part 1: Theoretical Foundations , 2017, IEEE Intelligent Systems.

[72]  Thorsten Joachims,et al.  Fairness of Exposure in Rankings , 2018, KDD.

[73]  Krishna P. Gummadi,et al.  Blind Justice: Fairness with Encrypted Sensitive Attributes , 2018, ICML.

[74]  Donghee Shin,et al.  Role of fairness, accountability, and transparency in algorithmic affordance , 2019, Comput. Hum. Behav..

[75]  Kristina Lerman,et al.  A Survey on Bias and Fairness in Machine Learning , 2019, ACM Comput. Surv..

[76]  Lu Zhang,et al.  Situation Testing-Based Discrimination Discovery: A Causal Inference Approach , 2016, IJCAI.

[77]  Ryen W. White Belief dynamics in web search , 2014, J. Assoc. Inf. Sci. Technol..

[78]  Yiqun Liu,et al.  Fairness-Aware Group Recommendation with Pareto-Efficiency , 2017, RecSys.

[79]  Jon M. Kleinberg,et al.  Inherent Trade-Offs in the Fair Determination of Risk Scores , 2016, ITCS.

[80]  Duane Szafron,et al.  Visual Explanation of Evidence with Additive Classifiers , 2006, AAAI.

[81]  David Lazer,et al.  Auditing Autocomplete: Suggestion Networks and Recursive Algorithm Interrogation , 2019, WebSci.

[82]  Lu Zhang,et al.  A Causal Framework for Discovering and Removing Direct and Indirect Discrimination , 2016, IJCAI.

[83]  Jude W. Shavlik,et al.  Using Sampling and Queries to Extract Rules from Trained Neural Networks , 1994, ICML.

[84]  Ross B. Girshick,et al.  Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[85]  Bernard J. Jansen,et al.  Detecting Demographic Bias in Automatically Generated Personas , 2019, CHI Extended Abstracts.

[86]  Nathan Srebro,et al.  Equality of Opportunity in Supervised Learning , 2016, NIPS.

[87]  Minas Gjoka,et al.  Walking in Facebook: A Case Study of Unbiased Sampling of OSNs , 2010, 2010 Proceedings IEEE INFOCOM.

[88]  Emilee J. Rader,et al.  Explanations as Mechanisms for Supporting Algorithmic Transparency , 2018, CHI.

[89]  Olcay Boz,et al.  Extracting decision trees from trained neural networks , 2002, KDD.

[90]  Jun Zhao,et al.  'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions , 2018, CHI.

[91]  Ingmar Weber,et al.  The demographics of web search , 2010, SIGIR.

[92]  Panagiotis Papapetrou,et al.  A peek into the black box: exploring classifiers by randomization , 2014, Data Mining and Knowledge Discovery.

[93]  Terris Wolff Be careful... , 1998, SIGU.

[94]  Junghoo Cho,et al.  Impact of search engines on page popularity , 2004, WWW '04.

[95]  David Lazer,et al.  Auditing Partisan Audience Bias within Google Search , 2018, Proc. ACM Hum. Comput. Interact..

[96]  Mark Bilandzic,et al.  Bringing Transparency Design into Practice , 2018, IUI.

[97]  Krishna P. Gummadi,et al.  Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment , 2016, WWW.

[98]  Sameep Mehta,et al.  Analyze, Detect and Remove Gender Stereotyping from Bollywood Movies , 2018, FAT.

[99]  Gianluca Demartini,et al.  Investigating User Perception of Gender Bias in Image Search: The Role of Sexism , 2018, SIGIR.

[100]  Franco Turini,et al.  Local Rule-Based Explanations of Black Box Decision Systems , 2018, ArXiv.

[101]  Alejandro Bellogín,et al.  Statistical biases in Information Retrieval metrics for recommender systems , 2017, Information Retrieval Journal.

[102]  Birger Hjørland,et al.  Domain Analysis in Information Science , 2010 .

[103]  Jahna Otterbacher,et al.  What Is Beautiful Continues to Be Good - People Images and Algorithmic Inferences on Physical Attractiveness , 2019, INTERACT.

[104]  Judith Masthoff,et al.  A Survey of Explanations in Recommender Systems , 2007, 2007 IEEE 23rd International Conference on Data Engineering Workshop.

[105]  Krishna P. Gummadi,et al.  Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media , 2017, CSCW.

[106]  Patrick Gallinari,et al.  Coping with the Document Frequency Bias in Sentiment Classification , 2012, ICWSM.

[107]  E. George,et al.  Making sense of a forest of treesH , 2007 .

[108]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[109]  Krishna P. Gummadi,et al.  Potential for Discrimination in Online Targeted Advertising , 2018, FAT.

[110]  Catherine Tucker,et al.  Algorithmic Bias : A Counterfactual Perspective , 2017 .

[111]  Lu Zhang,et al.  On Convexity and Bounds of Fairness-aware Classification , 2019, WWW.

[112]  Brent J. Hecht,et al.  Measuring the Importance of User-Generated Content to Search Engines , 2019, ICWSM.

[113]  Vasudeva Varma,et al.  Stereotypical Bias Removal for Hate Speech Detection Task using Knowledge-based Generalizations , 2019, WWW.

[114]  Ingmar Weber,et al.  Automated Hate Speech Detection and the Problem of Offensive Language , 2017, ICWSM.

[115]  G. Ridgeway Transparency, Statistics, and Justice System Knowledge Is Essential for Science of Risk Assessment , 2020, 2.1.

[116]  Sean A. Munson,et al.  Unequal Representation and Gender Stereotypes in Image Search Results for Occupations , 2015, CHI.

[117]  Anne Marie Piper,et al.  Addressing Age-Related Bias in Sentiment Analysis , 2018, CHI.

[118]  Dietmar Jannach,et al.  A systematic review and taxonomy of explanations in decision support and recommender systems , 2017, User Modeling and User-Adapted Interaction.

[119]  Pedro M. Domingos Knowledge Discovery Via Multiple Models , 1998, Intell. Data Anal..

[120]  Adam Tauman Kalai,et al.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings , 2016, NIPS.

[121]  Michael Luca,et al.  Racial Discrimination in the Sharing Economy: Evidence from a Field Experiment , 2016 .

[122]  Giles Hooker,et al.  Tree Space Prototypes: Another Look at Making Tree Ensembles Interpretable , 2016, FODS.

[123]  Toon Calders,et al.  Classifying without discriminating , 2009, 2009 2nd International Conference on Computer, Control and Communication.

[124]  Toon Calders,et al.  Three naive Bayes approaches for discrimination-free classification , 2010, Data Mining and Knowledge Discovery.

[125]  Krishna P. Gummadi,et al.  Who Makes Trends? Understanding Demographic Biases in Crowdsourced Recommendations , 2017, ICWSM.

[126]  Susan C. Herring,et al.  Cultural bias in Wikipedia content on famous persons , 2011, J. Assoc. Inf. Sci. Technol..

[127]  R. Caruana,et al.  Detecting Bias in Black-Box Models Using Transparent Model Distillation. , 2017 .

[128]  Toniann Pitassi,et al.  Fairness through awareness , 2011, ITCS '12.

[129]  Klaus-Robert Müller,et al.  Feature Importance Measure for Non-linear Learning Algorithms , 2016, ArXiv.

[130]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[131]  Beth St. Jean,et al.  There’s a creepy guy on the other end at Google!: engaging middle school students in a drawing activity to elicit their mental models of Google , 2017, Information Retrieval Journal.

[132]  Virgílio A. F. Almeida,et al.  A Framework for Benchmarking Discrimination-Aware Models in Machine Learning , 2019, AIES.

[133]  Elke A. Rundensteiner,et al.  FARE: Diagnostics for Fair Ranking using Pairwise Error Metrics , 2019, WWW.

[134]  David Lazer,et al.  Location, Location, Location: The Impact of Geolocation on Web Search Personalization , 2015, Internet Measurement Conference.

[135]  Amina Adadi,et al.  Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) , 2018, IEEE Access.

[136]  Nava Tintarev,et al.  SIREN: A Simulation Framework for Understanding the Effects of Recommender Systems in Online News Environments , 2019, FAT.

[137]  Jahna Otterbacher,et al.  What Makes an Image Tagger Fair? , 2019, UMAP.

[138]  D. Kupfer,et al.  The CAD-MDD : A Computerized Adaptive Diagnostic Screening Tool for Depression , 2013 .

[139]  Mike Thelwall,et al.  Are scholarly articles disproportionately read in their own country? An analysis of mendeley readers , 2015, J. Assoc. Inf. Sci. Technol..

[140]  Indre Zliobaite,et al.  A survey on measuring indirect discrimination in machine learning , 2015, ArXiv.

[141]  Mariusz Bojarski,et al.  VisualBackProp: visualizing CNNs for autonomous driving , 2016, ArXiv.

[142]  Bhaskar Mitra,et al.  An Eye-tracking Study of User Interactions with Query Auto Completion , 2014, CIKM.

[143]  Kripabandhu Ghosh,et al.  Fairness of Extractive Text Summarization , 2018, WWW.

[144]  Giovanni Quattrone,et al.  There's No Such Thing as the Perfect Map: Quantifying Bias in Spatial Crowd-sourcing Datasets , 2015, CSCW.

[145]  Nisheeth K. Vishnoi,et al.  Classification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees , 2018, FAT.

[146]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[147]  Andrea Vedaldi,et al.  Interpretable Explanations of Black Boxes by Meaningful Perturbation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[148]  Christoph Trattner,et al.  The impact of image descriptions on user tagging behavior: A study of the nature and functionality of crowdsourced tags , 2015, J. Assoc. Inf. Sci. Technol..

[149]  Iyad Rahwan,et al.  Darling or Babygirl ? Investigating Stylistic Bias in Sentiment Analysis , 2018 .

[150]  Jude W. Shavlik,et al.  in Advances in Neural Information Processing , 1996 .

[151]  Jonathan E. Fieldsend,et al.  Confident Interpretation of Bayesian Decision Tree Ensembles for Clinical Applications , 2007, IEEE Transactions on Information Technology in Biomedicine.

[152]  Cynthia Rudin,et al.  The age of secrecy and unfairness in recidivism prediction , 2018, 2.1.

[153]  Fernando Diaz,et al.  Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems , 2018, CIKM.

[154]  Susan Leavy,et al.  Gender Bias in Artificial Intelligence: The Need for Diversity and Gender Theory in Machine Learning , 2018, 2018 IEEE/ACM 1st International Workshop on Gender Equality in Software Engineering (GE).

[155]  Filip Radlinski,et al.  On user interactions with query auto-completion , 2014, SIGIR.

[156]  A. Chouldechova,et al.  Toward Algorithmic Accountability in Public Services: A Qualitative Study of Affected Community Perspectives on Algorithmic Decision-making in Child Welfare Services , 2019, CHI.

[157]  Franco Turini,et al.  A Survey of Methods for Explaining Black Box Models , 2018, ACM Comput. Surv..

[158]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[159]  F. Maxwell Harper,et al.  Exploring Question Selection Bias to Identify Experts and Potential Experts in Community Question Answering , 2012, TOIS.

[160]  Franco Turini,et al.  Integrating induction and deduction for finding evidence of discrimination , 2009, Artificial Intelligence and Law.