YoungSheldon at SemEval-2021 Task 5: Fine-tuning Pre-trained Language Models for Toxic Spans Detection using Token classification Objective

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 953–959 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 953 YoungSheldon at SemEval-2021 Task 5: Fine-tuning Pre-trained Language Models for Toxic Spans Detection using Token classification Objective Mayukh Sharma, Ilanthenral Kandasamy, W.B. Vasantha School of Computer Science and Engineering Vellore Institute of Technology Vellore, Tamil Nadu, India 04mayukh@gmail.com,ilanthenral.k@vit.ac.in, vasantha.wb@vit.ac.in Abstract In this paper, we describe our system used for SemEval 2021 Task 5: Toxic Spans De- tection. Our proposed system approaches the problem as a token classification task. We trained our model to find toxic words and con- catenate their spans to predict the toxic spans within a sentence. We fine-tuned Pre-trained Language Models (PLMs) for identifying the toxic words. For fine-tuning, we stacked the classification layer on top of the PLM fea- tures of each word to classify if it is toxic or not. PLMs are pre-trained using different ob- jectives and their performance may differ on downstream tasks. We, therefore, compare the performance of BERT, ELECTRA, RoBERTa, XLM-RoBERTa, T5, XLNet, and MPNet for identifying toxic spans within a sentence. Our best performing system used RoBERTa. It per- formed well, achieving an F1 score of 0.6841 and secured a rank of 16 on the official leader- board. 1 Introduction Internet and social networking sites have brought people together by providing a simple yet effec- tive method of communication. Over the years people used it to exchange positive ideas but re- cently, there has been a rise in toxic content and hate speech over the internet (Zampieri et al., 2019, 2020). Most datasets (Fortuna et al., 2020) dealing with the problem of toxic, offensive, or hateful con- tent aim to classify the entire text belonging to a particular class. They do not identify the parts of the text that make it toxic. Manual filtering of toxic data is tough and can cause mental and emotional stress to annotators (Zampieri et al., 2019). An automatic system with the ability to identify toxic text and highlighting toxic spans can be useful for the moderators. It will help save time and prevent stress caused by reading long texts. SemEval 2021 Task 5: Toxic Spans Detection(Pavlopoulos et al., 2021) draws attention to the problem of identifying toxic spans present in a sentence. Our proposed system makes use of a word-level classifier for detecting the offensive words present in a sentence. The offsets of the toxic words can then be concatenated to find the toxic spans. We made use of pre-trained language models (PLMs) for building our classifier. We experimented with BERT(Devlin et al., 2019), ELECTRA(Clark et al., 2020), RoBERTa(Liu et al., 2019b), XLNet(Yang et al., 2020), MPNet(Song et al., 2020), T5(Raffel et al., 2020), and XLM-RoBERTa(Conneau et al., 2020) to compare their performance on the task of toxic spans detection. Owing to the increase in the number of pre-trained language models choosing the correct model is an important decision as these models contain millions of parameters and are ex- pensive …

Content truncated to 500 words