MIPT-NSU-UTMN at SemEval-2021 Task 5: Ensembling Learning with Pre-trained Language Models for Toxic Spans Detection

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 913–918 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 913 MIPT-NSU-UTMN at SemEval-2021 Task 5: Ensembling Learning with Pre-trained Language Models for Toxic Spans Detection Mikhail Kotyushev Moscow Institute of Physics and Technology Moscow, Russia mkotyushev@gmail.com Anna Glazkova University of Tyumen Tyumen, Russia a.v.glazkova@utmn.ru Dmitry Morozov Novosibirsk State University Novosibirsk, Russia morozowdm@gmail.com Abstract This paper describes our system for SemEval- 2021 Task 5 on Toxic Spans Detection. We de- veloped ensemble models using BERT-based neural architectures and post-processing to combine tokens into spans. We evaluated sev- eral pre-trained language models using various ensemble techniques for toxic span identifica- tion and achieved sizable improvements over our baseline fine-tuned BERT models. Finally, our system obtained a F1-score of 67.55% on test data. 1 Introduction Toxic speech has become a rising issue for social media communities. Abusive content is very di- verse and therefore offensive language and toxic speech detection is not a trivial issue. Besides, so- cial media moderation of lengthy comments and posts is often a time-consuming process. In this regard, the task of detecting toxic spans in social media texts deserves close attention. This work is based on the participation of our team, named MIPT-NSU-UTMN, in SemEval 2021 Task 5, “Toxic Spans Detection” (Pavlopoulos et al., 2021). Organizers of the shared task pro- vided participants with the trial, train, and test sets of English social media comments annotated at the span level indicating the presence or absence of text toxicity. We formulated the task as a token clas- sification problem and investigated several BERT- based models using two-step knowledge transfer. We found that preliminary fine-tuning of the model on data that is close to the target domain im- proves the quality of the token classification. The source code of our models is available at https: //github.com/morozowdmitry/semeval21. The paper is organized as follows. A brief review of related work is given in Section 2. The definition of the task has been summarized in Section 3. The proposed methods and experimental settings have been elaborated in Section 4. Section 5 contains the results and error analysis respectively. Section 6 is a conclusion. 2 Related Work Computational approaches to tackle text toxicity have recently gained a lot of interest due to the widespread use of social media. Since moderation is crucial to promoting healthy online discussions, research on toxicity detection has been attracting much attention. Our work is also related to hate speech and abusive language detection (Fortuna et al., 2020). The toxic speech detection task is usually framed as a supervised learning problem. Moreover, fairly generic features, such as bag of words (Harris, 1954) or word embeddings (Mikolov et al., 2013), systematically yield reasonable classification per- formance (Fortuna and Nunes, 2018; Schmidt and Wiegand, 2017). To better understand the mech- anisms of toxic speech detection, some scholars (Waseem et al., 2017; Lee et al., 2018; Karan and ˇSnajder, 2018; Swamy et al., 2019) compared different techniques for abusive …

Content truncated to 500 words