GUIR at SemEval-2020 Task 12: Domain-Tuned Contextualized Models for Offensive Language Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1555–1561 Barcelona, Spain (Online), December 12, 2020. 1555 GUIR at SemEval-2020 Task 12: Domain-Tuned Contextualized Models for Offensive Language Detection Sajad Sotudeh∗ †, Tong Xiang∗ ‡, Hao-Ren Yao†, Sean MacAvaney†, Eugene Yang†, Nazli Goharian†, Ophir Frieder† Information Retrieval Lab Georgetown University, USA †{firstname}@ir.cs.georgetown.edu ‡tx39@georgetown.edu Abstract Offensive language detection is an important and challenging task in natural language processing. We present our submissions to the OffensEval 2020 shared task, which includes three English sub-tasks: identifying the presence of offensive language (Sub-task A), identifying the presence of target in offensive language (Sub-task B), and identifying the categories of the target (Sub-task C). Our experiments explore using a domain-tuned contextualized language model (namely, BERT) for this task. We also experiment with different components and configurations (e.g., a multi-view SVM) stacked upon BERT models for specific sub-tasks. Our submissions achieve F1 scores of 91.7% in Sub-task A, 66.5% in Sub-task B, and 63.2% in Sub-task C. We perform an ablation study which reveals that domain tuning considerably improves the classification performance. Furthermore, error analysis shows common misclassification errors made by our model and outlines research directions for future. 1 Introduction The rapid development of user-generated content in social media has given millions of people the ability to easily share their ideas with each other. While users can publicly communicate their beliefs with others, their published content may be offensive to other individuals or groups. Since offensive speech can jeopardize others’ ability to express themselves, many social media platforms restrict the type of acceptable content on their platforms. Manually detecting such content is expensive and time-consuming. Thus, automatic detection of these behaviors in social media has attracted researchers’ attention. Although this topic has been explored in prior works (Yao, 2019; Rizos et al., 2019; Zhang et al., 2018; Nobata et al., 2016), the task of offensive language detection still remains challenging. The OffensEval 2020 shared task (Zampieri et al., 2020) aims to encourage continued work in this area, and we progress the study of offensive language detection through our participation in this shared task utilizing contextualized language models. OffensEval 2020 evaluates various aspects of offensive language following the scheme of the OLID dataset (Zampieri et al., 2019a), including identifying the presence of offensive language (Sub-task A), identifying whether the offensive language is targeted (Sub-task B), and identifying whether the target of the offensive language is an individual, group, or something else (Sub-task C). The task extends the prior work by introducing a new data collection from Twitter that spans multiple languages. Our experiments explore various versions of the BERT model (Bidirectional Encoder Representations from Transformers (Devlin et al., 2019)) tuned for offensive language identification. As previous studies have mentioned, dealing with social media content can be challenging because it is often short and noisy (Qian et al., 2018). To alleviate this problem, we first fine-tune a BERT model on a large amount of unlabelled data gathered from Twitter. We pre-process the tweets by splitting hashtags …

Content truncated to 500 words