UoB at SemEval-2021 Task 5: Extending Pre-Trained Language Models to Include Task and Domain-Specific Information for Toxic Span Prediction

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 243–248 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 243 UoB at SemEval-2021 Task 5: Extending Pre-Trained Language Models to Include Task and Domain-Specific Information for Toxic Span Prediction Erik Yan School of Computer Science University of Birmingham United Kingdom erikdyan@outlook.com Harish Tayyar Madabushi School of Computer Science University of Birmingham United Kingdom Harish@HarishTayyarMadabushi.com Abstract Toxicity is pervasive in social media and poses a major threat to the health of online commu- nities. The recent introduction of pre-trained language models, which have achieved state- of-the-art results in many NLP tasks, has trans- formed the way in which we approach natu- ral language processing. However, the inher- ent nature of pre-training means that they are unlikely to capture task-specific statistical in- formation or learn domain-specific knowledge. Additionally, most implementations of these models typically do not employ conditional random fields, a method for simultaneous to- ken classification. We show that these modifi- cations can improve model performance on the Toxic Spans Detection task at SemEval-2021 to achieve a score within 4 percentage points of the top performing team. 1 Introduction and Motivation Moderation is crucial to promoting healthy online discussions. The anonymity afforded by computer- mediated communication enables individuals to engage in toxic behaviour which they would oth- erwise not consider. Although many datasets and models focusing on toxicity detection have been released, most of them classify entire sequences of text, and do not highlight the individual words that make a text toxic. The Toxic Spans Detection task at SemEval-2021 (Pavlopoulos et al., 2021) focuses on the evaluation of systems that can accu- rately identify toxic spans within text. Highlighting such spans can provide more information to human moderators in the form of attribution, instead of an unexplained toxicity score per post, and is thus a crucial step towards successful semi-automated moderation. In this paper we focus on the shared task, wherein systems are expected to extract a list of toxic spans, or an empty list, per text. A toxic span is defined as a sequence of words that contributes to a text’s toxicity. Since 2018, NLP models have adopted the con- cept of generative pre-training on a diverse cor- pus of unlabelled text, followed by supervised fine- tuning on specific tasks (Radford et al., 2018). Pre- trained models are built to simulate anthropomor- phic learning, wherein existing knowledge can be adapted to new tasks without the need to train on these tasks from scratch - a requirement of tra- ditional machine learning models. This idea of transfer learning, whilst powerful, leads to models being fine-tuned on target tasks using significantly fewer epochs than was previously standard. This reduced training on the target task means that task- specific statistical information or domain-specific knowledge may not be learned by these models. Such task-specific data may include count-based information, which has been shown to improve the performance of pre-trained models in sequence classification tasks (Lim and Madabushi, 2020; Prakash and …

Content truncated to 500 words