hBERT + BiasCorp - Fighting Racism on the Web

Subtle and overt racism is still present both in physical and online communities today and has impacted many lives in different segments of the society. In this short piece of work, we present how we’re tackling this societal issue with Natural Language Processing. We are releasing BiasCorp, a dataset containing 139,090 comments and news segment from three specific sources - Fox News, BreitbartNews and YouTube. The first batch (45,000 manually annotated) is ready for publication. We are currently in the final phase of manually labeling the remaining dataset using Amazon Mechanical Turk. BERT has been used widely in several downstream tasks. In this work, we present hBERT, where we modify certain layers of the pretrained BERT model with the new Hopfield Layer. hBert generalizes well across different distributions with the added advantage of a reduced model complexity. We are also releasing a JavaScript library 3 and a Chrome Extension Application, to help developers make use of our trained model in web applications (say chat application) and for users to identify and report racially biased contents on the web respectively

[1]  Kevin Duh,et al.  Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning , 2020, RepL4NLP@ACL.

[2]  Julia Hirschberg,et al.  Detecting Hate Speech on the World Wide Web , 2012 .

[3]  Thomas Wolf,et al.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter , 2019, ArXiv.

[4]  Katharine Gelber,et al.  Hate Speech and Freedom of Speech in Australia , 2007 .

[5]  David Yarowsky,et al.  DECISION LISTS FOR LEXICAL AMBIGUITY RESOLUTION: Application to Accent Restoration in Spanish and French , 1994, ACL.

[6]  Geir Kjetil Sandve,et al.  Hopfield Networks is All You Need , 2020, ArXiv.

[7]  Joel R. Tetreault,et al.  Abusive Language Detection in Online User Content , 2016, WWW.

[8]  Alan F. Smeaton,et al.  Text Categorisation of Racist Texts Using a Support Vector Machine , 2004 .

[9]  Dirk Hovy,et al.  Demographic Factors Improve Classification Performance , 2015, ACL.

[10]  Abhinav Gupta,et al.  Learning from Noisy Large-Scale Datasets with Minimal Supervision , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Kevin W. Saunders What about Hate Speech , 2011 .

[12]  Bing Liu,et al.  Mining and summarizing customer reviews , 2004, KDD.

[13]  Moshe Wasserblat,et al.  Q8BERT: Quantized 8Bit BERT , 2019, 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS).

[14]  David Bamman,et al.  Distributed Representations of Geographically Situated Language , 2014, ACL.

[15]  Takuya Akiba,et al.  Optuna: A Next-generation Hyperparameter Optimization Framework , 2019, KDD.

[16]  Andy Way,et al.  Demographic Word Embeddings for Racism Detection on Twitter , 2017, IJCNLP.

[17]  Hind Saleh Alatawi,et al.  Detecting White Supremacist Hate Speech Using Domain Specific Word Embedding With Deep Learning and BERT , 2020, IEEE Access.

[18]  Pete Burnap,et al.  Us and them: identifying cyber hate on Twitter across multiple protected characteristics , 2016, EPJ Data Science.

[19]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.