Annotating Hate and Offenses on Social Media

This paper describes a corpus annotation process to support the identification of hate speech and offensive language in social media. In addition, we provide the first robust corpus this kind for the Brazilian Portuguese language. The corpus was collected from Instagram pages of political personalities and manually annotated, being composed by 7,000 documents annotated according to three different layers: a binary classification (offensive versus non-offensive language), the level of offense (highly offensive, moderately offensive and slightly offensive messages), and the identification regarding the target of the discriminatory content (xenophobia, racism, homophobia, sexism, religion intolerance, partyism, apology to the dictatorship, antisemitism and fat phobia). Each comment was annotated by three different annotators, which achieved high inter-annotator agreement. The proposed annotation approach is also language and domain independent, nevertheless, it was currently applied for Brazilian Portuguese.

[1]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[2]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[3]  B. Robinson,et al.  Fat phobia: measuring, understanding, and changing anti-fat attitudes. , 1993, The International journal of eating disorders.

[4]  Thomas H. Lewis,et al.  The Authoritarian Specter , 1997 .

[5]  William Julius Wilson,et al.  The Bridge over the Racial Divide: Rising Inequality and Coalition Politics , 1999 .

[6]  J. Sim,et al.  The kappa statistic in reliability studies: use, interpretation, and sample size requirements. , 2005, Physical therapy.

[7]  Winfried Brugger.  Proibição ou proteção do discurso do ódio? algumas observações sobre o direito alemão e o americano. , 2007 .

[8]  Kevin W. Saunders.  What about Hate Speech , 2011 .

[9]  Walter Daelemans,et al.  Detection and Fine-Grained Classification of Cyberbullying Events , 2015, RANLP.

[10]  Candace Moore,et al.  Kappa , 2015, Radiopaedia.org.

[11]  Shuhua Liu,et al.  Text Classification Models for Web Content Filtering and Online Safety , 2015, 2015 IEEE International Conference on Data Mining Workshop (ICDMW).

[12]  James D. Wright,et al.  Sociology of Racism , 2015 .

[13]  Kalliopi Chainoglou,et al.  European Union Agency for Fundamental Rights (FRA) , 2016 .

[14]  Dirk Hovy,et al.  Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter , 2016, NAACL.

[15]  Radhika Mamidi,et al.  When does a compliment become sexist? Analysis and classification of ambivalent sexism using twitter data , 2017, NLP+CSS@ACL.

[16]  Ralf Peters,et al.  Detecting Offensive Statements towards Foreigners in Social Media , 2017, HICSS.

[17]  Cody Buntain,et al.  A Large Labeled Corpus for Online Harassment Research , 2017, WebSci.

[18]  Walid Magdy,et al.  Abusive Language Detection on Arabic Social Media , 2017, ALW@ACL.

[19]  Michael Wiegand,et al.  A Survey on Hate Speech Detection using Natural Language Processing , 2017, SocialNLP@EACL.

[20]  Ingmar Weber,et al.  Automated Hate Speech Detection and the Problem of Offensive Language , 2017, ICWSM.

[21]  Lei Gao,et al.  Detecting Online Hate Speech Using Context Aware Models , 2017, RANLP.

[22]  Andy Way,et al.  Demographic Word Embeddings for Racism Detection on Twitter , 2017, IJCNLP.

[23]  Ika Alfina,et al.  Hate speech detection in the Indonesian language: A dataset and preliminary study , 2017, 2017 International Conference on Advanced Computer Science and Information Systems (ICACSIS).

[24]  John Pavlopoulos,et al.  Deep Learning for User Comment Moderation , 2017, ALW@ACL.

[25]  Rogers Prates de Pelle,et al.  Offensive Comments in the Brazilian Web: a dataset and baseline results , 2017 .

[26]  Stefaan Walgrave,et al.  The tie that divides: Cross‐national evidence of the primacy of partyism , 2018 .

[27]  Ona de Gibert,et al.  Hate Speech Dataset from a White Supremacy Forum , 2018, ALW.

[28]  Indra Budi,et al.  A Dataset and Preliminaries Study for Abusive Language Detection in Indonesian Social Media , 2018 .

[29]  Cristina Bosco,et al.  An Impossible Dialogue! Nominal Utterances and Populist Rhetoric in an Italian Twitter Corpus of Hate Speech against Immigrants , 2018, LREC.

[30]  Shivakant Mishra,et al.  International Conference on Advances in Social Networks Analysis and Mining ( ASONAM ) Are They Our Brothers ? Analysis and Detection of Religious Hate Speech in the Arabic Twittersphere , 2018 .

[31]  Paolo Rosso,et al.  Overview of the Task on Automatic Misogyny Identification at IberEval 2018 , 2018, IberEval@SEPLN.

[32]  Tomaž Erjavec,et al.  Datasets of Slovene and Croatian Moderated News Comments , 2018, ALW.

[33]  Sérgio Nunes,et al.  A Survey on Automatic Detection of Hate Speech in Text , 2018, ACM Comput. Surv..

[34]  Marco Guerini,et al.  CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech , 2019, ACL.

[35]  Damir Cavar,et al.  Annotating Antisemitic Online Content. Towards an Applicable Definition of Antisemitism , 2019, ArXiv.

[36]  Sandra Kübler,et al.  Investigating Multilingual Abusive Language Detection: A Cautionary Tale , 2019, RANLP.

[37]  Ricardo Ribeiro,et al.  Automatic cyberbullying detection: A systematic review , 2019, Comput. Hum. Behav..

[38]  Hugo Jair Escalante,et al.  Overview of MEX-A3T at IberLEF 2019: Authorship and Aggressiveness Analysis in Mexican Spanish Tweets , 2018, IberLEF@SEPLN.

[39]  Yangqiu Song,et al.  Multilingual and Multi-Aspect Hate Speech Analysis , 2019, EMNLP.

[40]  Sérgio Nunes,et al.  A Hierarchically-Labeled Portuguese Hate Speech Dataset , 2019, Proceedings of the Third Workshop on Abusive Language Online.

[41]  Paolo Rosso,et al.  SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter , 2019, *SEMEVAL.

[42]  L. Oliveira.  Imigrantes, xenofobia e racismo: uma análise de conflitos em escolas municipais de São Paulo , 2019 .

[43]  Francielle Alves Vargas,et al.  Identifying Fine-Grained Opinion and Classifying Polarity on Coronavirus Pandemic , 2020, BRACIS.

[44]  Zhiwei Gao,et al.  Offensive Language Detection on Video Live Streaming Chat , 2020, COLING.

[45]  Fabrício Benevenuto,et al.  Characterizing Toxicity on Facebook Comments in Brazil , 2020, WebMedia.

[46]  Marcos Zampieri,et al.  Offensive Language Identification in Greek , 2020, LREC.

[47]  Kalina Bontcheva,et al.  Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis , 2020, AACL.

[48]  Çağrı Çöltekin,et al.  A Corpus of Turkish Offensive Language on Social Media , 2020, LREC.