HITMI&T at SemEval-2021 Task 5: Integrating Transformer and CRF for Toxic Spans Detection

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 870–874 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 870 HITMI&T at SemEval-2021 Task 5: Integrating Transformer and CRF for Toxic Spans Detection Chenyi Wang∗, Tianshu Liu∗, Tiejun Zhao† Machine Intelligence and Translation Laboratory, Harbin Institute of Technology, Harbin, China wcy708708@126.com, liutianshu99@163.com, tjzhao@hit.edu.cn Abstract This paper introduces our system at SemEval- 2021 Task 5: Toxic Spans Detection. The task aims to accurately locate toxic spans within a text. Using BIO tagging scheme, we model the task as a token-level sequence labeling task. Our system uses a single model built on the model of multi-layer bidirectional transformer encoder. And we introduce conditional ran- dom field (CRF) to make the model learn the constraints between tags. We use ERNIE as pre-trained model, which is more suitable for the task accroding to our experiments. In addi- tion, we use adversarial training with the fast gradient method (FGM) to improve the robust- ness of the system. Our system obtains 69.85% F1 score, ranking 3rd for the official evalua- tion. 1 Introduction With the prosperity of the Internet, it is easier and easier for people to get information and publish their opinions online. However, sometimes users’ opinions can be offensive to others. Because toxic posts will have a negative impact on the network environment, and manual identification is time- consuming and expensive, automatic detection of these behaviors has attracted researchers’ attention. After adapting the hate-speech problem to the problem of word sense disambiguation, an ap- proach to detect hate speech in online text is pre- sented (Warner and Hirschberg, 2012), which uses template-based strategy to generate features and an SVM classifier to identify whether the text is toxic or not. In SemEval-2020 Task 12 (Zampieri et al., 2020) and SemEval-2019 Task 6 (Zampieri et al., 2019), which also related to offensive statements, ∗ Equal contribution. † Corresponding author. transformer-based methods were the most popular approaches for their great advantages in learning word representations in context. In Semeval-2021 task 5: Toxic Spans Detec- tion (Pavlopoulos et al., 2021), the organizers use posts from the publicly available Civil Comments dataset (Borkan et al., 2019), which already com- prises post-level toxicity annotations. After manual annotation, character-level annotation results are obtained, which are the toxic spans we need to lo- cate. The task extends the prior work by identifying spans that make a text toxic, which can better ex- plain why posts are offensive rather than just giving a system-generated unexplained toxicity score. We model the task as a sequence labeling task be- cause toxic spans are contextually influenced. Our model is in Transformer-CRF architecture, and we try different pre-trained models as the transformer’s initialization to fine-tune model suitable for toxic spans detection. The Conditional Random Fields (CRF) (Lafferty et al., 2001) allows the model to learn the constraints between tags. We also use the Fast Gradient Method (FGM) (Miyato et al., 2016) as adversarial training strategy, which ap- plies perturbation to word embedding to …

Content truncated to 500 words