The Arabic Parallel Gender Corpus 2.0: Extensions and Analyses

Gender bias in natural language processing (NLP) applications, particularly machine translation, has been receiving increasing attention. Much of the research on this issue has focused on mitigating gender bias in English NLP models and systems. Addressing the problem in poorly resourced, and/or morphologically rich languages has lagged behind, largely due to the lack of datasets and resources. In this paper, we introduce a new corpus for gender identification and rewriting in contexts involving one or two target users (I and/or You) – first and second grammatical persons with independent grammatical gender preferences. We focus on Arabic, a gender-marking morphologically rich language. The corpus has multiple parallel components: four combinations of 1st and 2nd person in feminine and masculine grammatical genders, as well as English, and English to Arabic machine translation output. This corpus expands on Habash et al. (2019)’s Arabic Parallel Gender Corpus (APGC v1.0) by adding second person targets as well as increasing the total number of sentences over 6.5 times, reaching over 590K words. Our new dataset will aid the research and development of gender identification, controlled text generation, and post-editing rewrite systems that could be used to personalize NLP applications and provide users with the correct outputs based on their grammatical gender preferences. We make the Arabic Parallel Gender Corpus (APGC v2.0) publicly available.1

[1]  Nanyun Peng,et al.  The Woman Worked as a Babysitter: On Biases in Language Generation , 2019, EMNLP.

[2]  Eva Vanmassenhove,et al.  gENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomena , 2021, GEBNLP.

[3]  Bill Byrne,et al.  First the worst: Finding better gender translations during beam search , 2021, ArXiv.

[4]  Jiliang Tang,et al.  Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learning , 2020, EMNLP.

[5]  Kemal Oflazer,et al.  Large Scale Arabic Error Annotation: Guidelines and Framework , 2014, LREC.

[6]  Lucia Specia,et al.  Personalized Machine Translation: Preserving Original Author Traits , 2016, EACL.

[7]  Yonatan Belinkov,et al.  Investigating Gender Bias in Language Models Using Causal Mediation Analysis , 2020, NeurIPS.

[8]  Marco Gaido,et al.  Gender Bias in Machine Translation , 2021, Transactions of the Association for Computational Linguistics.

[9]  Noah A. Smith,et al.  Evaluating Gender Bias in Machine Translation , 2019, ACL.

[10]  William Yang Wang,et al.  They, Them, Theirs: Rewriting with Gender-Neutral English , 2021, ArXiv.

[11]  Jiliang Tang,et al.  Does Gender Matter? Towards Fairness in Dialogue Systems , 2020, COLING.

[12]  Jörg Tiedemann,et al.  OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles , 2016, LREC.

[13]  Nizar Habash,et al.  On Arabic Transliteration , 2007 .

[14]  Malihe Alikhani,et al.  Examining Covert Gender Bias: A Case Study in Turkish and English Machine Translation Models , 2021, INLG.

[15]  Kellie Webster,et al.  Automatically Identifying Gender Issues in Machine Translation using Perturbations , 2020, EMNLP.

[16]  Jieyu Zhao,et al.  Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer , 2020, ACL.

[17]  Chris Emmery,et al.  NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender Neutral Alternatives , 2021, EMNLP.

[18]  Nizar Habash,et al.  An Arabic Morphological Analyzer and Generator with Copious Features , 2018 .

[19]  Nizar Habash,et al.  A Corpus for Modeling Morpho-Syntactic Agreement in Arabic: Gender, Number and Rationality , 2011, ACL.

[20]  Nizar Habash,et al.  Introduction to Arabic Natural Language Processing , 2010, Introduction to Arabic Natural Language Processing.

[21]  Shikha Bordia,et al.  Identifying and Reducing Gender Bias in Word-Level Language Models , 2019, NAACL.

[22]  Nizar Habash,et al.  Automatic Gender Identification and Reinflection in Arabic , 2019, Proceedings of the First Workshop on Gender Bias in Natural Language Processing.

[23]  Verena Rieser,et al.  Conversational Assistants and Gender Stereotypes: Public Perceptions and Desiderata for Voice Personas , 2020, GEBNLP.

[24]  Houda Bouamor,et al.  Gender-Aware Reinflection using Linguistically Enhanced Neural Models , 2020, GEBNLP.

[25]  Ryan Cotterell,et al.  It’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution , 2019, EMNLP.

[26]  Michela Menegatti,et al.  Gender Bias and Sexism in Language , 2017 .

[27]  B. Byrne,et al.  Neural Machine Translation Doesn’t Translate Gender Coreference Right Unless You Make It , 2020, GEBNLP.

[28]  Ryan Cotterell,et al.  Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology , 2019, ACL.

[29]  Mai ElSherief,et al.  Mitigating Gender Bias in Natural Language Processing: Literature Review , 2019, ACL.

[30]  Kai-Wei Chang,et al.  “Nice Try, Kiddo”: Investigating Ad Hominems in Dialogue Responses , 2021, NAACL.

[31]  Matt Post,et al.  A Call for Clarity in Reporting BLEU Scores , 2018, WMT.

[32]  Marcis Pinnis,et al.  Mitigating Gender Bias in Machine Translation with Target Gender Annotations , 2020, WMT.

[33]  Nanyun Peng,et al.  Revealing Persona Biases in Dialogue Systems , 2021, ArXiv.

[34]  Yoav Goldberg,et al.  Filling Gender & Number Gaps in Neural Machine Translation with Black-box Context Injection , 2019, Proceedings of the First Workshop on Gender Bias in Natural Language Processing.

[35]  Jason Weston,et al.  Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation , 2020, EMNLP.

[36]  Yang Trista Cao,et al.  Toward Gender-Inclusive Coreference Resolution , 2019, ACL.

[37]  Jieyu Zhao,et al.  Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods , 2018, NAACL.

[38]  Zeyu Li,et al.  Learning Gender-Neutral Word Embeddings , 2018, EMNLP.

[39]  Yoav Goldberg,et al.  Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them , 2019, NAACL-HLT.

[40]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[41]  Marta R. Costa-jussà,et al.  Equalizing Gender Bias in Neural Machine Translation with Word Embeddings Techniques , 2019, Proceedings of the First Workshop on Gender Bias in Natural Language Processing.

[42]  Solon Barocas,et al.  Language (Technology) is Power: A Critical Survey of “Bias” in NLP , 2020, ACL.

[43]  Rachel Rudinger,et al.  Gender Bias in Coreference Resolution , 2018, NAACL.

[44]  Alan W Black,et al.  Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings , 2019, NAACL.

[45]  Alexander Erdmann,et al.  CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing , 2020, LREC.

[46]  Andy Way,et al.  Getting Gender Right in Neural Machine Translation , 2019, EMNLP.

[47]  M. Costa-jussà,et al.  Fine-tuning Neural Machine Translation on Gender-Balanced Datasets , 2020, GEBNLP.

[48]  Bill Byrne,et al.  Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem , 2020, ACL.

[49]  Nizar Habash,et al.  MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic , 2014, LREC.

[50]  Anupam Datta,et al.  Gender Bias in Neural Natural Language Processing , 2018, Logic, Language, and Security.

[51]  Adam Tauman Kalai,et al.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings , 2016, NIPS.

[52]  Ahmed Y. Tawfik,et al.  Gender aware spoken language translation applied to English-Arabic , 2018, 2018 2nd International Conference on Natural Language and Speech Processing (ICNLSP).