Detecting and Monitoring Hate Speech in Twitter

Social Media are sensors in the real world that can be used to measure the pulse of societies. However, the massive and unfiltered feed of messages posted in social media is a phenomenon that nowadays raises social alarms, especially when these messages contain hate speech targeted to a specific individual or group. In this context, governments and non-governmental organizations (NGOs) are concerned about the possible negative impact that these messages can have on individuals or on the society. In this paper, we present HaterNet, an intelligent system currently being used by the Spanish National Office Against Hate Crimes of the Spanish State Secretariat for Security that identifies and monitors the evolution of hate speech in Twitter. The contributions of this research are many-fold: (1) It introduces the first intelligent system that monitors and visualizes, using social network analysis techniques, hate speech in Social Media. (2) It introduces a novel public dataset on hate speech in Spanish consisting of 6000 expert-labeled tweets. (3) It compares several classification approaches based on different document representation strategies and text classification models. (4) The best approach consists of a combination of a LTSM+MLP neural network that takes as input the tweet’s word, emoji, and expression tokens’ embeddings enriched by the tf-idf, and obtains an area under the curve (AUC) of 0.828 on our dataset, outperforming previous methods presented in the literature.

[1]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[2]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[3]  Andrew B. Whinston,et al.  Designing a social-broadcasting-based business intelligence system , 2011, TMIS.

[4]  H. Varian,et al.  Predicting the Present with Google Trends , 2009 .

[5]  Munmun De Choudhury,et al.  Analyzing the Dynamics of Communication in Online Social Networks , 2010, Handbook of Social Network Technologies.

[6]  Zeerak Waseem,et al.  Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter , 2016, NLP+CSS@EMNLP.

[7]  Bernard J. Jansen,et al.  Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for Identifying and Classifying Hate in Online News Media , 2018, ICWSM.

[8]  Paolo Rosso,et al.  Detecting Deceptive Opinions: Intra and Cross-Domain Classification Using an Efficient Representation , 2017, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[9]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[10]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[11]  M. Kaminski.  The right to explanation, explained , 2018, Research Handbook on Information Law and Governance.

[12]  Vivek Narayanan,et al.  Fast and Accurate Sentiment Classification Using an Enhanced Naive Bayes Model , 2013, IDEAL.

[13]  Johan Bollen,et al.  Twitter mood predicts the stock market , 2010, J. Comput. Sci..

[14]  Felice Dell'Orletta,et al.  Hate Me, Hate Me Not: Hate Speech Detection on Facebook , 2017, ITASEC.

[15]  Dirk Neumann,et al.  Crime Mapping through Geo-Spatial Social Media Activity , 2014, ICIS.

[16]  Michael I. Jordan,et al.  Machine learning: Trends, perspectives, and prospects , 2015, Science.

[17]  Björn Gambäck,et al.  Using Convolutional Neural Networks to Classify Hate-Speech , 2017, ALW@ACL.

[18]  A. Downs.  Up and Down with Ecology--The Issue Attention Cycle , 1972 .

[19]  Kai Wu,et al.  Social Media as Sensor in Real World: Geolocate User with Microblog , 2014, NLPCC.

[20]  Fabrício Benevenuto,et al.  Analyzing the Targets of Hate in Online Social Media , 2016, ICWSM.

[21]  Panagiotis Takis Metaxas,et al.  How (Not) to Predict Elections , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[22]  Harith Alani,et al.  Evaluation Datasets for Twitter Sentiment Analysis: A survey and a new dataset, the STS-Gold , 2013, ESSEM@AI*IA.

[23]  Helmut Schmidt,et al.  Probabilistic part-of-speech tagging using decision trees , 1994 .

[24]  Wenpu Xing,et al.  Weighted PageRank algorithm , 2004, Proceedings. Second Annual Conference on Communication Networks and Services Research, 2004..

[25]  Efthimios Tambouris,et al.  Understanding the Predictive Power of Social Media This is a pre-print version of the following article : , 2013 .

[26]  Eric P. Xing,et al.  Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2014, ACL 2014.

[27]  Matthew S. Gerber,et al.  Predicting crime using Twitter and kernel density estimation , 2014, Decis. Support Syst..

[28]  Sérgio Nunes,et al.  A Survey on Automatic Detection of Hate Speech in Text , 2018, ACM Comput. Surv..

[29]  David Robinson,et al.  Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network , 2018, ESWC.

[30]  Walter Daelemans,et al.  Pattern for Python , 2012, J. Mach. Learn. Res..

[31]  Haojie Zhu,et al.  A Spatio-Temporal Kernel Density Estimation Framework for Predictive Crime Hotspot Mapping and Evaluation , 2018, Applied Geography.

[32]  Ingmar Weber,et al.  Automated Hate Speech Detection and the Problem of Offensive Language , 2017, ICWSM.

[33]  Xinyu Chen,et al.  Crime prediction using Twitter sentiment and weather , 2015, 2015 Systems and Information Engineering Design Symposium.

[34]  Connie St Louis,et al.  Can Twitter predict disease outbreaks? , 2012, BMJ : British Medical Journal.

[35]  Huan Liu,et al.  Feature Selection and Classification - A Probabilistic Wrapper Approach , 1996, IEA/AIE.

[36]  Spencer Ch.  The Utility of Hotspot Mapping for Predicting Spatial Patterns of Crime , 2008 .

[37]  Wei Ding,et al.  Crime Forecasting Using Data Mining Techniques , 2011, 2011 IEEE 11th International Conference on Data Mining Workshops.

[38]  Ron Kohavi,et al.  Feature Selection for Knowledge Discovery and Data Mining , 1998 .

[39]  Ronald Rousseau,et al.  Social network analysis: a powerful strategy, also for the information sciences , 2002, J. Inf. Sci..

[40]  Claire Cardie,et al.  Negative Deceptive Opinion Spam , 2013, NAACL.

[41]  Björn Ross,et al.  Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis , 2016, ArXiv.

[42]  Esra Sahin,et al.  Spam/ham e-mail classification using machine learning methods based on bag of words technique , 2018, 2018 26th Signal Processing and Communications Applications Conference (SIU).

[43]  Claire Cardie,et al.  Finding Deceptive Opinion Spam by Any Stretch of the Imagination , 2011, ACL.

[44]  Pascale Fung,et al.  One-step and Two-step Classification for Abusive Language Detection on Twitter , 2017, ALW@ACL.

[45]  M. McHugh.  Interrater reliability: the kappa statistic , 2012, Biochemia medica.

[46]  J. Fernando Sánchez-Rada,et al.  Enhancing deep learning sentiment analysis with ensemble techniques in social applications , 2020 .

[47]  A. Smeaton,et al.  On Using Twitter to Monitor Political Sentiment and Predict Election Results , 2011 .

[48]  Matthew S. Gerber,et al.  Forecasting Violent Extremist Cyber Recruitment , 2015, IEEE Transactions on Information Forensics and Security.

[49]  Deepjyoti Choudhury,et al.  COMMUNITY DETECTION IN SOCIAL NETWORKS: AN OVERVIEW , 2013 .

[50]  Yejin Choi,et al.  The Risk of Racial Bias in Hate Speech Detection , 2019, ACL.

[51]  Xiaofeng Wang,et al.  Spatio-temporal modeling of criminal incidents using geographic, demographic, and twitter-derived information , 2012, 2012 IEEE International Conference on Intelligence and Security Informatics.

[52]  Michelle R. Guy,et al.  Twitter earthquake detection: earthquake monitoring in a social world , 2012 .

[53]  Marek R. Ogiela,et al.  Clustering of trending topics in microblogging posts: A graph-based approach , 2017, Future Gener. Comput. Syst..

[54]  Kang Liu,et al.  Book Review: Sentiment Analysis: Mining Opinions, Sentiments, and Emotions by Bing Liu , 2015, CL.

[55]  Matt J. Kusner,et al.  From Word Embeddings To Document Distances , 2015, ICML.

[56]  Alex Graves,et al.  Supervised Sequence Labelling with Recurrent Neural Networks , 2012, Studies in Computational Intelligence.

[57]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[58]  Carter C. Price,et al.  Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations , 2013 .

[59]  Federico Liberatore,et al.  A Decision Support System for predictive police patrolling , 2015, Decis. Support Syst..

[60]  Min Song,et al.  Opinion polarity detection in Twitter data combining shrinkage regression and topic modeling , 2016, J. Informetrics.

[61]  Paolo Rosso,et al.  Detection of Opinion Spam with Character n-grams , 2015, CICLing.

[62]  Serkan Günal,et al.  A novel probabilistic feature selection method for text classification , 2012, Knowl. Based Syst..

[63]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[64]  Hang-Bong Kang,et al.  Prediction of crime occurrence from multi-modal data using deep learning , 2017, PloS one.

[65]  H. Varian,et al.  Predicting the Present with Google Trends , 2012 .

[66]  Fabio Franch.  (Wisdom of the Crowds)2: 2010 UK Election Prediction with Social Media , 2013 .

[67]  Dirk Hovy,et al.  Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter , 2016, NAACL.

[68]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[69]  L. Ungar,et al.  Data-Driven Content Analysis of Social Media , 2015 .

[70]  Claire Cardie,et al.  Towards a General Rule for Identifying Deceptive Opinion Spam , 2014, ACL.

[71]  Eric Gilbert,et al.  VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text , 2014, ICWSM.

[72]  Liang-Chih Yu,et al.  Grid-Based Crime Prediction Using Geographical Features , 2018, ISPRS Int. J. Geo Inf..

[73]  Yuichi Sasaki,et al.  The anatomy of tweet overload: How number of tweets received, number of friends, and egocentric network density affect perceived information overload , 2015, Telematics Informatics.

[74]  Florian Probst,et al.  Online social networks: A survey of a global phenomenon , 2012 .

[75]  José Camacho-Collados,et al.  Applying automatic text-based detection of deceptive language to police reports: Extracting behavioral patterns from a multi-step classification model to understand how we lie to the police , 2018, Knowl. Based Syst..

[76]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[77]  Marcin Mironczuk,et al.  A recent overview of the state-of-the-art elements of text classification , 2018, Expert Syst. Appl..

[78]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[79]  Carlo Strapparava,et al.  The Lie Detector: Explorations in the Automatic Recognition of Deceptive Language , 2009, ACL.

[80]  Yun Zhu,et al.  Support vector machines and Word2vec for text classification with semantic features , 2015, 2015 IEEE 14th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC).

[81]  Aron Culotta,et al.  Towards detecting influenza epidemics by analyzing Twitter messages , 2010, SOMA '10.

[82]  Derechos Internacionales y Exteriores.  Office of the United Nations High Commissioner for Human Rights , 2010 .

[83]  Jing Zhou,et al.  Hate Speech Detection with Comment Embeddings , 2015, WWW.

[84]  J. A. Bolúfer,et al.  Diccionario de la lengua española , 1917 .

[85]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[86]  George E. Tita,et al.  Self-Exciting Point Process Modeling of Crime , 2011 .

[87]  Vasudeva Varma,et al.  Deep Learning for Hate Speech Detection in Tweets , 2017, WWW.

[88]  Wu He,et al.  Social Media-Based Forecasting: A Case Study of Tweets and Stock Prices in the Financial Services Industry , 2016, J. Organ. End User Comput..

[89]  Wilpen L. Gorr,et al.  Leading Indicators and Spatial Interactions: A Crime‐Forecasting Model for Proactive Police Deployment , 2007 .