UPB at SemEval-2021 Task 5: Virtual Adversarial Training for Toxic Spans Detection

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 225–232 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 225 UPB at SemEval-2021 Task 5: Virtual Adversarial Training for Toxic Spans Detection Andrei Paraschiv, Dumitru-Clementin Cercel, Mihai Dascalu University Politehnica of Bucharest, Faculty of Automatic Control and Computers andrei.paraschiv74@stud.acs.upb.ro {dumitru.cercel, mihai.dascalu}@upb.ro Abstract The real-world impact of polarization and tox- icity in the online sphere marked the end of 2020 and the beginning of this year in a nega- tive way. Semeval-2021, Task 5 - Toxic Spans Detection is based on a novel annotation of a subset of the Jigsaw Unintended Bias dataset and is the first language toxicity detection task dedicated to identifying the toxicity-level spans. For this task, participants had to auto- matically detect character spans in short com- ments that render the message as toxic. Our model considers applying Virtual Adversarial Training in a semi-supervised setting during the fine-tuning process of several Transformer- based models (i.e., BERT and RoBERTa), in combination with Conditional Random Fields. Our approach leads to performance improve- ments and more robust models, enabling us to achieve an F1-score of 65.73% in the official submission and an F1-score of 66.13% after further tuning during post-evaluation. 1 Introduction Nowadays, online engagement in social activities is at its highest levels. The lockdowns during the 2020 COVID-19 pandemic increased the overall time spent online. In Germany for instance, Leme- nager et al. (2021) observed that 71% of considered subjects increased their online media consumption during this period. Unfortunately, online toxicity is present in a large part of the social and news media platforms. As such, automated early detection is necessary since toxic behavior is often contagious and leads to a spillover effect (Kwon and Gruzd, 2017). Recently, a significant effort was put into the detection of toxic and offensive language (van Aken et al., 2018; Paraschiv and Cercel, 2019; Tanase et al., 2020b,a), but the challenging na- ture of these problems leaves several avenues un- explored. In addition, most shared tasks focus on the distinction between toxic/non-toxic (Wul- czyn et al., 2017; van Aken et al., 2018; Juuti et al., 2020) or offensive/non-offensive posts in var- ious languages (Struß et al., 2019; Zampieri et al., 2019a,b, 2020; Mandl et al., 2020; Arag´on et al., 2020). The Semeval-2021 Task 5, namely Toxic Spans Detection (Pavlopoulos et al., 2021), tack- les the problem of identifying the exact portion of the document that gives it toxicity. The provided dataset is a subset of the Jigsaw Unintended Bias in Toxicity Classification dataset1, with annotated spans that represent toxicity from a document. In this paper, we describe our participation in the aforementioned Toxic Spans Detection task using several Transformer-based models (Vaswani et al., 2017), including BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), with a Conditional Random Field (CRF) (Lafferty et al., 2001) layer on top to identify spans that include toxic language. We introduce Virtual Adversarial Training (VAT) (Miyato et al., 2015) in our training …

Content truncated to 500 words