IITK@Detox at SemEval-2021 Task 5: Semi-Supervised Learning and Dice Loss for Toxic Spans Detection

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 211–219 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 211 IITK@Detox at SemEval-2021 Task 5: Semi-Supervised Learning and Dice Loss for Toxic Spans Detection Archit Bansal Abhay Kaushik Ashutosh Modi Indian Institute of Technology Kanpur (IIT Kanpur) {architb, kabhay}@iitk.ac.in ashutoshm@cse.iitk.ac.in Abstract In this work, we present our approach and findings for SemEval-2021 Task 5 - Toxic Spans Detection. The task’s main aim was to identify spans to which a given text’s tox- icity could be attributed. The task is chal- lenging mainly due to two constraints: the small training dataset and imbalanced class distribution. Our paper investigates two tech- niques, semi-supervised learning and learn- ing with Self-Adjusting Dice Loss, for tack- ling these challenges. Our submitted system (ranked ninth on the leader board) consisted of an ensemble of various pre-trained Trans- former Language Models trained using either of the above-proposed techniques. 1 Introduction Content moderation has become the topic of most conversations regarding social media platforms. However, with over 4 billion active internet users, it is impossible to moderate each piece of message generated online manually. Therefore, the focus is now shifting towards tackling the issue using machine learning methods. Various toxicity detection datasets (Wulczyn et al., 2017; Borkan et al., 2019) and models (Pavlopoulos et al., 2017; Liu et al., 2019; Seganti et al., 2019) have been successfully developed over the years to tackle the issue of moderation. How- ever, these have mostly focused on identifying whole comments or documents as either toxic or not. In semi-automated settings, a model merely generating a toxicity score for each comment, some of which can be very lengthy, is not of much help to human moderators. To tackle this issue, the Se- mEval 2021 Task 5 : Toxic Spans Detection is intro- duced (Pavlopoulos et al., 2021). The task involves identifying text spans in a given toxic post that con- tributes towards the toxicity of that post. The task aims to promote the development of a system that would augment human moderators by giving them more insights into what actually contributes to the text’s toxicity. The task is challenging mainly due to the follow- ing reasons: a) small size of the dataset b) character- istics of text samples extracted from social media leading to difficulties such as out-of-vocabulary words and ungrammatical sentences c) class im- balance in the dataset d) inconsistencies in data annotations. We approached this task as a sub- token level sequence labeling task. Fine-tuned pre- trained transformer language models (Qiu et al., 2020) are the backbone of all our approaches. We investigated two main techniques to enhance the re- sults of the fine-tuned transformer models, namely Semi-Supervised Learning (Yarowsky, 1995; Liu et al., 2011) and fine-tuning with Self-Adjusting Dice Loss (Li et al., 2020). This paper reports the results of our experiments with these different tech- niques and pre-trained transformer models. Our submitted system consisted of an ensemble of dif- ferent pre-trained transformer models and …

Content truncated to 500 words