The research described in this paper concerns automatic cyberbullying detection in social media. There are two goals to achieve: building a gold standard cyberbullying detection dataset and measuring the performance of the Samurai cyberbullying detection system. The Formspring dataset provided in a Kaggle competition was re-annotated as a part of the research. The annotation procedure is described in detail and, unlike many other recent data annotation initiatives, does not use Mechanical Turk for finding people willing to perform the annotation. The new annotation compared to the old one seems to be more coherent since all tested cyberbullying detection system performed better on the former. The performance of the Samurai system is compared with 5 commercial systems and one well-known machine learning algorithm, used for classifying textual content, namely Fasttext. It turns out that Samurai scores the best in all measures (accuracy, precision and recall), while Fasttext is the second-best performing algorithm.
[1]
Radha Poovendran,et al.
Deceiving Google's Perspective API Built for Detecting Toxic Comments
,
2017,
ArXiv.
[2]
Gary Marcus,et al.
Deep Learning: A Critical Appraisal
,
2018,
ArXiv.
[3]
Kelly Reynolds,et al.
Using Machine Learning to Detect Cyberbullying
,
2011,
2011 10th International Conference on Machine Learning and Applications and Workshops.
[4]
Tomas Mikolov,et al.
Bag of Tricks for Efficient Text Classification
,
2016,
EACL.
[5]
Jacob Cohen,et al.
The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability
,
1973
.