Automatic Hate Speech Detection using Machine Learning: A Comparative Study

The increasing use of social media and information sharing has given major benefits to humanity. However, this has also given rise to a variety of challenges including the spreading and sharing of hate speech messages. Thus, to solve this emerging issue in social media sites, recent studies employed a variety of feature engineering techniques and machine learning algorithms to automatically detect the hate speech messages on different datasets. However, to the best of our knowledge, there is no study to compare the variety of feature engineering techniques and machine learning algorithms to evaluate which feature engineering technique and machine learning algorithm outperform on a standard publicly available dataset. Hence, the aim of this paper is to compare the performance of three feature engineering techniques and eight machine learning algorithms to evaluate their performance on a publicly available dataset having three distinct classes. The experimental results showed that the bigram features when used with the support vector machine algorithm best performed with 79% off overall accuracy. Our study holds practical implication and can be used as a baseline study in the area of detecting automatic hate speech messages. Moreover, the output of different comparisons will be used as state-of-art techniques to compare future researches for existing automated text classification techniques.

[1]  Mohammad Ghodsi,et al.  Comparison of artificial neural network and logistic regression models for prediction of mortality in head trauma based on initial clinical data , 2005, BMC Medical Informatics Decis. Mak..

[2]  Pete Burnap,et al.  Us and them: identifying cyber hate on Twitter across multiple protected characteristics , 2016, EPJ Data Science.

[3]  Joel R. Tetreault,et al.  Abusive Language Detection in Online User Content , 2016, WWW.

[4]  Pierre Zweigenbaum,et al.  Text mining for pharmacovigilance: Using machine learning for drug name recognition and drug-drug interaction extraction and classification , 2015, J. Biomed. Informatics.

[5]  Sérgio Nunes,et al.  A Survey on Automatic Detection of Hate Speech in Text , 2018, ACM Comput. Surv..

[6]  Lucila Ohno-Machado,et al.  A Comparison of Machine Learning Methods for the Diagnosis of Pigmented Skin Lesions , 2001, J. Biomed. Informatics.

[7]  Robert E. Schapire,et al.  The Boosting Approach to Machine Learning An Overview , 2003 .

[8]  Jörg Becker,et al.  Discussing the Value of Automatic Hate Speech Detection in Online Debates , 2018 .

[9]  Sotiris B. Kotsiantis,et al.  Machine learning: a review of classification and combining techniques , 2006, Artificial Intelligence Review.

[10]  M. W Gardner,et al.  Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciences , 1998 .

[11]  Njagi Dennis Gitari,et al.  A Lexicon-based Approach for Hate Speech Detection , 2015, MUE 2015.

[12]  Tao Yang,et al.  Word Embedding for Understanding Natural Language: A Survey , 2018 .

[13]  Shuhua Liu,et al.  Combining N-gram based Similarity Analysis with Sentiment Analysis in Web Content Classification , 2014, KDIR.

[14]  Juan Enrique Ramos,et al.  Using TF-IDF to Determine Word Relevance in Document Queries , 2003 .

[15]  Vandana,et al.  Survey of Nearest Neighbor Techniques , 2010, ArXiv.

[16]  Leon Derczynski,et al.  Offensive Language and Hate Speech Detection for Danish , 2019, LREC.

[17]  Yunming Ye,et al.  An Improved Random Forest Classifier for Text Categorization , 2012, J. Comput..

[18]  Michael Wiegand,et al.  A Survey on Hate Speech Detection using Natural Language Processing , 2017, SocialNLP@EACL.

[19]  Ying Cao,et al.  Advance and Prospects of AdaBoost Algorithm , 2013, ACTA AUTOMATICA SINICA.

[20]  David D. Lewis,et al.  Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval , 1998, ECML.

[21]  Shervin Malmasi,et al.  Detecting Hate Speech in Social Media , 2017, RANLP.

[22]  Dirk Hovy,et al.  Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter , 2016, NAACL.

[23]  Taghi M. Khoshgoftaar,et al.  A Study on the Relationships of Classifier Performance Metrics , 2009, 2009 21st IEEE International Conference on Tools with Artificial Intelligence.

[24]  Mi Lu,et al.  Comparisons and Selections of Features and Classifiers for Short Text Classification , 2017 .

[25]  Henry Lieberman,et al.  Modeling the Detection of Textual Cyberbullying , 2011, The Social Mobile Web.

[26]  Yuzhou Wang,et al.  Locate the Hate: Detecting Tweets against Blacks , 2013, AAAI.

[27]  Pravesh Kumar Singh,et al.  Methodological Study Of Opinion Mining And Sentiment Analysis Techniques , 2014 .

[28]  Y. Ho,et al.  Simple Explanation of the No-Free-Lunch Theorem and Its Implications , 2002 .

[29]  Manish Shrivastava,et al.  Degree based Classification of Harmful Speech using Twitter Data , 2018, TRAC@COLING 2018.

[30]  W. B. Cavnar,et al.  N-gram-based text categorization , 1994 .

[31]  Ying Chen.  DETECTING OFFENSIVE LANGUAGE IN SOCIAL MEDIAS FOR PROTECTION OF ADOLESCENT ONLINE SAFETY , 2011 .

[32]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[33]  Liyana Shuib,et al.  Prediction of cause of death from forensic autopsy reports using text classification techniques: A comparative study. , 2017, Journal of forensic and legal medicine.

[34]  Walter Daelemans,et al.  A Dictionary-based Approach to Racism Detection in Dutch Social Media , 2016, ArXiv.

[35]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[36]  Yarimar Bonilla,et al.  Deprovincializing Trump, decolonizing diversity, and unsettling anthropology , 2017 .

[37]  Yunming Ye,et al.  An improved random forest classifier for image classification , 2012, 2012 IEEE International Conference on Information and Automation.

[38]  Igi Ardiyanto,et al.  Text Classification to Detect Student Level of Understanding in Prior Knowledge Activation Process , 2017 .

[39]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[40]  Alan F. Smeaton,et al.  Classifying racist texts using a support vector machine , 2004, SIGIR '04.

[41]  Nazli Goharian,et al.  Hate speech detection: Challenges and solutions , 2019, PloS one.

[42]  Zhi-Hua Zhou,et al.  A k-nearest neighbor based algorithm for multi-label classification , 2005, 2005 IEEE International Conference on Granular Computing.