SRPOL DIALOGUE SYSTEMS at SemEval-2021 Task 5: Automatic Generation of Training Data for Toxic Spans Detection

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 974–983 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 974 SRPOL DIALOGUE SYSTEMS at SemEval-2021 Task 5: Automatic Generation of Training Data for Toxic Spans Detection Michał Satława, Katarzyna Zamły´nska, Jarosław Piersa, Joanna Kolis, Klaudia Firl ˛ag, Katarzyna Beksa. Zuzanna Bordzicka, Christian Goltz, Paweł Bujnowski Samsung R&D Institute Poland {m.satlawa;k.zamlynska;j.piersa}@samsung.com joanna.kolis@gmail.com, {k.firlag;k.beksa}@samsung.com {z.bordzicka;c.goltz;p.bujnowski}@samsung.com Piotr Andruszkiewicz Samsung R&D Institute Poland Warsaw University of Technology p.andruszki2@samsung.com Abstract This paper presents a system used for SemEval-2021 Task 5: Toxic Spans Detection. Our system is an ensemble of BERT-based mo- dels for binary word classification, trained on a dataset extended by toxic comments modi- fied and generated by two language models. For the toxic word classification, the predic- tion threshold value was optimized separately for every comment, in order to maximize the expected F1 value. 1 Introduction Freedom of speech is one of the most important human rights. However, because the definition of protected speech is not precise enough, it can be easily misinterpreted and misused. The problem is magnified in cyberspace, where anonymity and asynchronous communication contribute to toxic disinhibition. As a result, the Internet has become space where hatred, harsh criticism, rude language and threats may grow. Currently, identification of such harmful content may depend mostly upon classification models that detect abusive comments or documents. However, SemEval-2021 Task 5: Toxic Spans Detection pro- poses detecting fragments of text that make it toxic, with the aim of supporting manual moderation of oftentimes lengthy comments. A successful solu- tion to this problem would be a crucial step towards more constructive and inclusive online discussions. The task focuses on English, which is the most com- mon language used on the Internet, as of January 2020 (Johnson, 2021). In this paper we present the model we used for toxic span detection, the method we used to find optimal prediction threshold values, and two me- thods for producing new training examples with toxic spans annotation: • Resampling the data: new examples are ge- nerated by substituting non-toxic words with predictions from a language model. • Data generation: we trained a simple language model on existing examples, in order to gene- rate new examples containing marked toxic spans. A total of 91 teams made an official submission on the test set with the best submission achieving F1 score of 0.7083. Our approach was ranked as 11th with 0.6865 F1 score (for details see Section A of the Appendix). 2 Related Work The interest in automatic identification of abusive language has increased among researchers due to the importance of public discussion in the Internet and its public impact. To our mind, the domain overlaps with other NLP tasks, such as sentiment analysis or comment classification. In an earlier piece of work, (Yin et al., 2009) stu- died harassment detection on Web 2.0 datasets (i.e. Kongregate, Slashdot and MySpace) using TFIDF n-gram features and SVM models. In other rese- arch, (Sood et al., 2012a,b) analysed profanity …

Content truncated to 500 words