MITRE at SemEval-2019 Task 5: Transfer Learning for Multilingual Hate Speech Detection

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 453–459 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 453 MITRE at SemEval-2019 Task 5: Transfer Learning for Multilingual Hate Speech Detection Abigail S. Gertner, John C. Henderson, Amy Marsh, Elizabeth M. Merkhofer, Ben Wellner and Guido Zarrella The MITRE Corporation 202 Burlington Road Bedford, MA 01730-1420, USA {gertner,jhndrsn,amarsh,emerkhofer,wellner,jzarrella}@mitre.org Abstract This paper describes MITRE’s participation in SemEval-2019 Task 5, HatEval: Multilin- gual detection of hate speech against immi- grants and women in Twitter. The techniques explored range from simple bag-of-ngrams classifiers to neural architectures with varied attention mechanisms. We describe several styles of transfer learning from auxiliary tasks, including a novel method for adapting pre- trained BERT models to Twitter data. Logis- tic regression ties the systems together into an ensemble submitted for evaluation. The result- ing system was used to produce predictions for all four HatEval subtasks, achieving the best mean rank of all teams that participated in all four conditions. 1 Introduction The popularity of social media allows anyone to post their thoughts and opinions for all to see. While the vast majority of these communications are benign, there are those who express hateful or threatening messages online. The identification of hate speech (Fortuna and Nunes, 2018; Schmidt and Wiegand, 2017) on platforms like Twitter is of particular interest for law enforcement and to social media companies who wish to remove ac- counts with offending content from their sites. Au- tomating the identification of hate speech will al- low platforms to flag and remove content much more quickly and effectively. In this effort we explored neural transfer learn- ing techniques, including word embeddings and fine-tuning of models trained with diverse auxil- iary tasks. We built and compared models em- ploying soft attention over sequences and multi- headed self-attention. We also present a novel task to aid in performing additional pre-training of BERT (Devlin et al., 2018) for domain adaptation to Twitter data. 2 Task, Data and Evaluation HatEval was a shared task organized within SemEval-2019 (Basile et al., 2019). The pri- mary task was detection of hate speech in Twit- ter, specifically against immigrants and women. This multilingual shared task was organized into two sub-tasks, each presented in both English and Spanish, for a total of four sub-task evaluations. Task A The first sub-task was simply to identify tweets containing hate speech against immigrants or women. The official metric used for this binary classification task was macro-averaged F1 score, in which the F1 scores are calculated for both the positive hate speech and negative not hate speech classes and then those two scores are averaged. Task B The second sub-task involved the de- tection of two specific aspects of hate speech: whether it is targeted at an individual vs. a group of people, and whether it expresses aggression on the part of the author. In this annotation scheme, there is a dependency between these two cate- gories and the hate speech label used in …

Content truncated to 500 words