UNBNLP at SemEval-2019 Task 5 and 6: Using Language Models to Detect Hate Speech and Offensive Language

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 514–518 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 514 UNBNLP at SemEval-2019 Task 5 and 6: Using Language Models to Detect Hate Speech and Offensive Language Ali Hakimi Parizi, Milton King and Paul Cook Faculty of Computer Science, University of New Brunswick Fredericton, NB E3B 5A3, Canada ahakimi@unb.ca, milton.king@unb.ca, paul.cook@unb.ca Abstract In this paper we apply a range of approaches to language modeling — including word- level n-gram and neural language models, and character-level neural language models — to the problem of detecting hate speech and of- fensive language. Our findings indicate that language models are able to capture knowl- edge of whether text is hateful or offensive. However, our findings also indicate that more- conventional approaches to text classification often perform similarly or better. 1 Introduction SemEval 2019 Task 5 focuses on detecting hate speech in social media text, while Task 6 considers identifying offensive language. Despite the dif- ferences between hate speech and offensive lan- guage, each of these tasks can be viewed as a bi- nary classification problem. In each case, gold- standard training data is provided on which super- vised approaches can be trained. In this paper we consider whether approaches to classification based on language models — in- cluding word- and character-level neural language models, as well as more-conventional (word-level) n-gram language models — are able to distinguish between hateful and not hateful, and offensive and not offensive, language. We find that these ap- proaches outperform a most-frequent class base- line, indicating that language models can capture some knowledge of whether text is hateful or of- fensive. However, for task 5 — for which the testing data was made available for follow up ex- periments — we also find that more-conventional approaches to supervised classification, such as naive Bayes and fastText (Joulin et al., 2017), of- ten give similar or better results. 2 Related Work Social media such as Twitter and Facebook are widely used, and hate speech has become preva- lent in these platforms. In response to this, a sub- stantial amount of research has recently focused on detecting such disturbing comments. Prior work on hate speech and offensive lan- guage detection has mostly focused on supervised machine learning techniques (Mathur et al., 2018; Davidson et al., 2017). A recent shared task on identifying aggression in social media (Kumar et al., 2018) observed there is no significant differ- ence between the performance of neural networks and linear classifiers. Furthermore, this shared task received just one lexicon-based approach, and its performance was not promising. Moreover, all of these approaches required expensive feature en- gineering and pre-processing. Instead of engineering specific features to use in supervised classifiers, we can instead employ language models to model the type of text we want to detect. Language models have previously been applied for the purpose of text classification (e.g., Bai et al., 2004; Howard and Ruder, 2018). Among different types of language models, recurrent …

Content truncated to 500 words