Cross-lingual Zero- and Few-shot Hate Speech Detection Utilising Frozen Transformer Language Models and AXEL

Detecting hate speech, especially in low-resource languages, is a non-trivial challenge. To tackle this, we developed a tailored architecture based on frozen, pre-trained Transformers to examine cross-lingual zero-shot and few-shot learning, in addition to uni-lingual learning, on the HatEval challenge data set. With our novel attention-based classification block AXEL, we demonstrate highly competitive results on the English and Spanish subsets. We also re-sample the English subset, enabling additional, meaningful comparisons in the future.

[1]  Petra Kralj Novak,et al.  Embeddia at SemEval-2019 Task 6: Detecting Hate with Neural Network and Transfer Learning Approaches , 2019, *SEMEVAL.

[2]  Paolo Rosso,et al.  SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter , 2019, *SEMEVAL.

[3]  Torsten Zesch,et al.  LTL-UDE at SemEval-2019 Task 6: BERT and Two-Vote Classification for Categorizing Offensiveness , 2019, *SEMEVAL.

[4]  David Robinson,et al.  Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network , 2018, ESWC.

[5]  Masaki Aono,et al.  KDEHatEval at SemEval-2019 Task 5: A Neural Network Model for Detecting Hate Speech in Twitter , 2019, SemEval@NAACL-HLT.

[6]  Kathleen McKeown,et al.  Predictive Embeddings for Hate Speech Detection on Twitter , 2018, ALW.

[7]  Manri Cheon,et al.  MAMNet: Multi-path adaptive modulation network for image super-resolution , 2020, Neurocomputing.

[8]  Björn W. Schuller,et al.  Augment to Prevent: Short-Text Data Augmentation in Deep Learning for Hate-Speech Classification , 2019, CIKM.

[9]  Jian Zhu,et al.  UM-IU@LING at SemEval-2019 Task 6: Identifying Offensive Tweets Using BERT and SVMs , 2019, *SEMEVAL.

[10]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[11]  Hervé Jégou,et al.  Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion , 2018, EMNLP.

[12]  Regina Barzilay,et al.  Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing , 2019, NAACL.

[13]  Aliza Sarlan,et al.  Twitter sentiment analysis , 2014, Proceedings of the 6th International Conference on Information Technology and Multimedia.

[14]  Kyomin Jung,et al.  Comparative Studies of Detecting Abusive Language on Twitter , 2018, ALW.

[15]  David Robinson,et al.  Hate Speech Detection on Twitter: Feature Engineering v.s. Feature Selection , 2018, ESWC.

[16]  Miriam Benballa,et al.  Saagie at Semeval-2019 Task 5: From Universal Text Embeddings and Classical Features to Domain-specific Text Classification , 2019, SemEval@NAACL-HLT.

[17]  John C. Henderson,et al.  MITRE at SemEval-2019 Task 5: Transfer Learning for Multilingual Hate Speech Detection , 2019, SemEval@NAACL-HLT.

[18]  Luke S. Zettlemoyer,et al.  Dissecting Contextual Word Embeddings: Architecture and Representation , 2018, EMNLP.

[19]  Jie Li,et al.  Channel-Wise and Spatial Feature Modulation Network for Single Image Super-Resolution , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[20]  Indra Budi,et al.  Multi-label Hate Speech and Abusive Language Detection in Indonesian Twitter , 2019, Proceedings of the Third Workshop on Abusive Language Online.

[21]  John Pavlopoulos,et al.  ConvAI at SemEval-2019 Task 6: Offensive Language Identification and Categorization with Perspective and BERT , 2019, *SEMEVAL.

[22]  Sérgio Nunes,et al.  A Hierarchically-Labeled Portuguese Hate Speech Dataset , 2019, Proceedings of the Third Workshop on Abusive Language Online.

[23]  Rico Sennrich,et al.  Neural Machine Translation of Rare Words with Subword Units , 2015, ACL.

[24]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[25]  Ingmar Weber,et al.  Automated Hate Speech Detection and the Problem of Offensive Language , 2017, ICWSM.

[26]  Ona de Gibert,et al.  Hate Speech Dataset from a White Supremacy Forum , 2018, ALW.

[27]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[28]  Alex Nikolov,et al.  Nikolov-Radivchev at SemEval-2019 Task 6: Offensive Tweet Classification with BERT and Ensembles , 2019, *SEMEVAL.

[29]  Nazli Goharian,et al.  Hate speech detection: Challenges and solutions , 2019, PloS one.

[30]  Prakhar Gupta,et al.  Learning Word Vectors for 157 Languages , 2018, LREC.

[31]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[32]  Jun-Hyuk Kim,et al.  RAM: Residual Attention Module for Single Image Super-Resolution , 2018, ArXiv.

[33]  Noah A. Smith,et al.  To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks , 2019, RepL4NLP@ACL.

[34]  Mohammad Shoeybi,et al.  Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism , 2019, ArXiv.

[35]  Felice Dell'Orletta,et al.  Overview of the EVALITA 2018 Hate Speech Detection Task , 2018, EVALITA@CLiC-it.

[36]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[37]  Karsten Müller,et al.  Fanning the Flames of Hate: Social Media and Hate Crime , 2020, Journal of the European Economic Association.

[38]  Holger Schwenk,et al.  Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond , 2018, Transactions of the Association for Computational Linguistics.

[39]  Mark Dredze,et al.  Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT , 2019, EMNLP.

[40]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[41]  Guillaume Lample,et al.  Word Translation Without Parallel Data , 2017, ICLR.

[42]  Preslav Nakov,et al.  SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) , 2019, *SEMEVAL.

[43]  Anders Søgaard,et al.  A Survey of Cross-lingual Word Embedding Models , 2017, J. Artif. Intell. Res..

[44]  Dirk Hovy,et al.  Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter , 2016, NAACL.

[45]  Ralf Krestel,et al.  Challenges for Toxic Comment Classification: An In-Depth Error Analysis , 2018, ALW.

[46]  Zeerak Waseem,et al.  Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter , 2016, NLP+CSS@EMNLP.

[47]  Liang Zou,et al.  NULI at SemEval-2019 Task 6: Transfer Learning for Offensive Language Detection using Bidirectional Transformers , 2019, *SEMEVAL.

[48]  Jianming Wang,et al.  BNU-HKBU UIC NLP Team 2 at SemEval-2019 Task 6: Detecting Offensive Language Using BERT model , 2019, *SEMEVAL.

[49]  Xuejie Zhang,et al.  YNU-HPCC at SemEval-2019 Task 6: Identifying and Categorising Offensive Language on Twitter , 2019, SemEval@NAACL-HLT.

[50]  Graham Neubig,et al.  Cross-Lingual Word Embeddings for Low-Resource Language Modeling , 2017, EACL.

[51]  Guillaume Lample,et al.  Cross-lingual Language Model Pretraining , 2019, NeurIPS.

[52]  Sérgio Nunes,et al.  A Survey on Automatic Detection of Hate Speech in Text , 2018, ACM Comput. Surv..

[53]  Ziqi Zhang,et al.  Hate Speech Detection: A Solved Problem? The Challenging Case of Long Tail on Twitter , 2018, Semantic Web.

[54]  L. Shanley,et al.  On Cybersecurity, Crowdsourcing, and Social Cyber-Attack , 2013 .