Kungfupanda at SemEval-2020 Task 12: BERT-Based Multi-TaskLearning for Offensive Language Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 2060–2066 Barcelona, Spain (Online), December 12, 2020. 2060 Kungfupanda at SemEval-2020 Task 12: BERT-Based Multi-Task Learning for Offensive Language Detection Wenliang Dai*, Tiezheng Yu*, Zihan Liu, Pascale Fung Center for Artificial Intelligence Research (CAiRE) Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong {wdaiai,tyuah,zliucr}@connect.ust.hk, pascale@ece.ust.hk Abstract Nowadays, offensive content in social media has become a serious problem, and automatically detecting offensive language is an essential task. In this paper, we build an offensive language detection system, which combines multi-task learning with BERT-based models. Using a pre- trained language model such as BERT, we can effectively learn the representations for noisy text in social media. Besides, to boost the performance of offensive language detection, we leverage the supervision signals from other related tasks. In the OffensEval-2020 competition, our model achieves 91.51% F1 score in English Sub-task A, which is comparable to the first place (92.23% F1). An empirical analysis is provided to explain the effectiveness of our approaches. 1 Introduction Nowadays, offensive content has invaded social media and becomes a serious problem for government organizations, online communities, and social media platforms. Therefore, it is essential to automatically detect and throttle the offensive content before it appears in social media. Previous studies have investi- gated different aspects of offensive languages such as abusive language (Nobata et al., 2016; Mubarak et al., 2017) and hate speech (Malmasi and Zampieri, 2017; Davidson et al., 2017). Recently, (Zampieri et al., 2019a) first studied the target of the offensive language in twitter and (Zampieri et al., 2020) expand it into the multilingual version, which is practical for studying hate speech concerning a specific target. The task is based on a three-level hierarchical annotation schema that en- compasses the following three general sub-tasks: (A) Offensive Language Detection; (B) Categorization of Offensive Language; (C) Offensive Language Target Identification. To tackle this problem, we emphasize that it is crucial to leverage pre-trained language model (e.g., BERT (Devlin et al., 2018)) to better understand the meaning of sentences and generate expressive word- level representations due to the inherent data noise (e.g., misspelling, grammatical mistakes) in social media (e.g., twitter). In addition, we hypothesize that the internal connections exist among the three general sub-tasks, and to improve one task, we can leverage the information of the other two tasks. Therefore, we first generate the representations of the input text based on the pre-trained language model BERT, and then we conduct multi-task learning based on the representations. Experimental results show that leveraging more task information can improve the offensive language detection performance. In the OffensEval-2020 competition, our system achieves 91.51% macro-F1 score in English Sub-task A (ranked 7th out of 85 submissions). Especially, only the OLID (Zampieri et al., 2019a) is used to train our model and no additional data is used. Our code is available at: https: //github.com/wenliangdai/multi-task-offensive-language-detection. 2 Related Works In general, offensive language detection includes some particular types, such …

Content truncated to 500 words