Proposed Taxonomy for Gender Bias in Text; A Filtering Methodology for the Gender Generalization Subtype

The purpose of this paper is to present an empirical study on gender bias in text. Current research in this field is focused on detecting and correcting for gender bias in existing machine learning models rather than approaching the issue at the dataset level. The underlying motivation is to create a dataset which could enable machines to learn to differentiate bias writing from non-bias writing. A taxonomy is proposed for structural and contextual gender biases which can manifest themselves in text. A methodology is proposed to fetch one type of structural gender bias, Gender Generalization. We explore the IMDB movie review dataset and 9 different corpora from Project Gutenberg. By filtering out irrelevant sentences, the remaining pool of candidate sentences are sent for human validation. A total of 6123 judgments are made on 1627 sentences and after a quality check on randomly selected sentences we obtain an accuracy of 75%. Out of the 1627 sentences, 808 sentence were labeled as Gender Generalizations. The inter-rater reliability amongst labelers was of 61.14%.

[1]  Es Unterhalter,et al.  Measuring Gender inequality and equality in education , 2015 .

[2]  Gadi Gilam,et al.  The dark side of gendered language: The masculine-generic form as a cause for self-report bias. , 2015, Psychological assessment.

[3]  Yoav Goldberg,et al.  Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them , 2019, NAACL-HLT.

[4]  Jieyu Zhao,et al.  Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods , 2018, NAACL.

[5]  Marilyn Schwartz,et al.  Guidelines for bias-free writing , 1995 .

[6]  Daniel Jurafsky,et al.  Word embeddings quantify 100 years of gender and ethnic stereotypes , 2017, Proceedings of the National Academy of Sciences.

[7]  J C Winck,et al.  Times they are a-changing. , 2010, Revista portuguesa de pneumologia.

[8]  Luke S. Zettlemoyer,et al.  End-to-end Neural Coreference Resolution , 2017, EMNLP.

[9]  Saif Mohammad,et al.  Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems , 2018, *SEMEVAL.

[10]  M. Z. Corbett.  Clearing the air: some thoughts on gender-neutral writing , 1990 .

[11]  Thomas Newkirk,et al.  Misreading Masculinity: Speculations on the Great Gender Gap in Writing , 2000, Language Arts.

[12]  Jason Baldridge,et al.  Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns , 2018, TACL.

[13]  Adam Tauman Kalai,et al.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings , 2016, NIPS.

[14]  Deborah Cameron,et al.  Language, Gender, and Sexuality: Current Issues and New Directions , 2005 .

[15]  Mounia Lalmas,et al.  First Women, Second Sex: Gender Bias in Wikipedia , 2015, HT.

[16]  Latanya Sweeney,et al.  Discrimination in online ad delivery , 2013, CACM.

[17]  Rachel Rudinger,et al.  Gender Bias in Coreference Resolution , 2018, NAACL.

[18]  J. Ho,et al.  Bias at the intersection of identity: Conflicting social stereotypes of gender and race augment the perceived femininity and interpersonal warmth of smiling Black women , 2018 .

[19]  Raewyn Connell,et al.  Gender Reckonings: New Social Theory and Research , 2018 .

[20]  Christopher Potts,et al.  Learning Word Vectors for Sentiment Analysis , 2011, ACL.

[21]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[22]  Zeerak Waseem,et al.  Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter , 2016, NLP+CSS@EMNLP.

[23]  Jill Zarestky,et al.  Gender-Inclusive Educational Programs for Workforce Development , 2018 .

[24]  Zeyu Li,et al.  Learning Gender-Neutral Word Embeddings , 2018, EMNLP.

[25]  Ingrid Robeyns.  When will society be gender just , 2007 .