sthruggle at SemEval-2019 Task 5: An Ensemble Approach to Hate Speech Detection

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 484–488 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 484 sthruggle at SemEval-2019 Task 5: An Ensemble Approach to Hate Speech Detection Aria Nourbakhsh, Frida Vermeer, Gijs Wiltvank, Rob van der Goot University of Groningen Groningen, the Netherlands {a.nourbakhsh, f.h.vermeer, g.g.wiltvank}@student.rug.nl r.van.der.goot@rug.nl Abstract In this paper, we present our approach to de- tection of hate speech against women and im- migrants in tweets for our participation in the SemEval-2019 Task 5. We trained an SVM and an RF classifier using character bi- and trigram features and a BiLSTM pre-initialized with external word embeddings. We combined the predictions of the SVM, RF and BiLSTM in two different ensemble models. The first was a majority vote of the binary values, and the second used the average of the confidence scores. For development, we got the highest accuracy (75%) by the final ensemble model with majority voting. For testing, all models scored substantially lower and the scores be- tween the classifiers varied more. We believe that these large differences between the higher accuracies in the development phase and the lower accuracies we obtained in the testing phase have partly to do with differences be- tween the training, development and testing data. 1 Introduction An unwanted phenomenon that can be found across social media, is the publication of texts with hateful content. In SemEval-2019 Task 5 (Basile et al., 2019), it is defined as any communication that disparages a person or group. We focused on immigrants and women, which are two of the most targeted groups of people who are victims to this kind of discourse. The micro-blogging service Twitter is a medium on which posts containing hateful content can be found in abundance. In order to filter out tweets with such content, machine learning and neural network techniques can be used to discriminate tweets which do contain hate speech from tweets which do not. Among the characteristics of Twit- ter data we can point out its noisiness and the use of emojis and hashtags, which can be taken into account for the classification task. In this paper, we tried to solve this problem for English, by incorporating a variety of classi- fication algorithms including Support Vector Ma- chine (SVM), Random Forest (RF), and Bidirec- tional Long Short-Term Memory (BiLSTM). For the classical machine learning classification algo- rithms (SVM and RF) we used character n-grams and for the BiLSTM we used word embeddings trained on a huge amount of Twitter data. By com- bining these three models into an ensemble learn- ing model, we achieved our best results on the de- velopment data, so we submitted this model for this shared task. We start the paper by discussing some earlier work done in this field. Then, we describe the dataset we used for this task. In chapter 4, we present our approach. In chapter 5, we continue with the results of our methods and the discussion. Finally, we end with …

Content truncated to 500 words