S-NLP at SemEval-2021 Task 5: An Analysis of Dual Networks for Sequence Tagging

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 888–897 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 888 S-NLP at SemEval-2021 Task 5: An Analysis of Dual Networks for Sequence Tagging Viet Anh Nguyen ∗, Tam Minh Nguyen *, Huy Quang Dao *, Quang Huu Pham * AI Research Team R&D Lab, Sun Asterisk Inc. {nguyen.viet.anh-d, nguyen.minh.tam-b, dao.quang.huy-b, pham.huu.quang} @sun-asterisk.com Abstract The SemEval 2021 task 5: Toxic Spans Detec- tion is a task of identifying considered-toxic spans in text, which provides a valuable, au- tomatic tool for moderating online contents. This paper represents the second-place method for the task, an ensemble of two approaches. While one approach relies on combining differ- ent embedding methods to extract diverse se- mantic and syntactic representations of words in context; the other utilizes extra data with a slightly customized Self-training, a semi- supervised learning technique, for sequence tagging problems. Both of our architectures take advantage of a strong language model, which was fine-tuned on a toxic classification task. Although experimental evidence indi- cates higher effectiveness of the first approach than the second one, combining them leads to our best results of 70.77 F1-score on the test dataset. 1 Introduction Social Network sites are an integral part of our society. These platforms are often designed to max- imize user interaction without sufficient means to moderate such interactions. The amount of users being cyber-bullied by toxic comments has reached an alarming proportion (Chan et al., 2021). To effi- ciently maintain the health of online communities, an automatic online-content filtering tool needs to be developed. Numerous previous attempts to re- solve this issue have focused on toxic comment classification (Georgakopoulos et al., 2018; Chu et al., 2017; Pham et al., 2020; Risch and Krestel, 2020). Although these classification models are capable of detecting toxic comments, their outputs are not interpretable (Mathew et al., 2020). On the other hand, Toxic Spans Detection (Pavlopoulos et al., 2021) is a task of locating toxic ∗equal contribution segments in texts. With such a system, the moder- ators can easily highlight offensive words in com- ments, which is an essential and explainable assis- tance for automated comment rating. In this paper, we propose our two approaches to resolve the task. Our contributions are as follows: • We investigate the effectiveness of our slightly customized Self-training (Wei et al., 2021) technique for a sequence tagging problem - Toxic Spans Detection. • We explore the benefits of combining differ- ent word representations including Byte Pair Encoding (Sennrich et al., 2015), contextual character-level (Akbik et al., 2018), FastText (Bojanowski et al., 2016) and RoBERTa (Liu et al., 2019) word embeddings in order to uti- lize different syntactic and semantic informa- tion learned by these embedding methods. • Taking advantage of a well-domain-adaptive pre-trained language model on a classifica- tion task (Unbiased-toxic-RoBERTa (Hanu and Unitary team, 2020)), we successfully in- tegrate our two above-mentioned methods to achieve a high F1-score of 70.77 and rank 2nd at the Semeval 2021 Task …

Content truncated to 500 words