UM-IU@LING at SemEval-2019 Task 6: Identifying Offensive Tweets Using BERT and SVMs

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 788–795 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 788 UM-IU@LING at SemEval-2019 Task 6: Identifying Offensive Tweets Using BERT and SVMs Jian Zhu Department of Linguistics University of Michigan Ann Arbor, MI, USA lingjzhu@umich.edu Zuoyu Tian Department of Linguistics Indiana University Bloomington, IN, USA zuoytian@iu.edu Sandra K¨ubler Department of Linguistics Indiana University Bloomington, IN, USA skuebler@indiana.edu Abstract This paper describes the UM-IU@LING’s sys- tem for the SemEval 2019 Task 6: OffensEval. We take a mixed approach to identify and cate- gorize hate speech in social media. In subtask A, we fine-tuned a BERT based classifier to detect abusive content in tweets, achieving a macro F1 score of 0.8136 on the test data, thus reaching the 3rd rank out of 103 submissions. In subtasks B and C, we used a linear SVM with selected character n-gram features. For subtask C, our system could identify the tar- get of abuse with a macro F1 score of 0.5243, ranking it 27th out of 65 submissions. 1 Introduction With the increased influence of social media on modern society, large amounts of user-generated content emerge on the internet. Besides the ex- change of ideas, we also see an exponential in- crease of aggressive and potentially harmful con- tent, for example, hate speech. If we consider the amount of user-generated data, it is impractical to manually identify the malicious speech. Thus we need to develop methods to detect offensive speech automatically through computational mod- els. However, this task is challenging because nat- ural language is fraught with ambiguities, and lan- guage in social media is extremely noisy. Here we present our method to automatically identify- ing offensive content in tweets. We primarily focus on detecting whether a tweet contains offensive content or not (subtask A), and then determining the target of the offen- sive content (subtask C). For subtask A, we use pre-trained word embeddings by fine-tuning the BERT model (Devlin et al., 2018) for detecting of- fensive tweets. For subtasks B and C, BERT did not perform well, either because of limited train- ing data or because we did not find the appropriate hyperparameters. Thus we use an SVM classifier with character n-grams as features. We acciden- tally flipped the predicted labels in our submission to subtask B, which is why we do not report results of subtask B here. Among all teams participating in OffensEval, our models ranks 3rd out of 103 on subtask A and 27th out of 65 on subtask C. (see Zampieri et al., 2019b). 2 Related Work Detecting offensive language online is becom- ing more and more important (Schmidt and Wie- gand, 2017; Founta et al., 2018; Malmasi and Zampieri, 2018). To build an effective classi- fier, one of the major problems is to find the ap- propriate features. Normally, two types of fea- tures are utilized: surface features like n-grams and word representations trained by neural net- work. Most offensive language classifiers are trained …

Content truncated to 500 words