YNU-HPCC at SemEval-2021 Task 5: Using a Transformer-based Model with Auxiliary Information for Toxic Span Detection

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 841–845 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 841 YNU-HPCC at SemEval-2021 Task 5: Using a Transformer-based Model with Auxiliary Information for Toxic Span Detection Ruijun Chen, Jin Wang and Xuejie Zhang School of Information Science and Engineering Yunnan University Kunming, China Contact: chenrj@mail.ynu.edu.cn, {wangjin, xjzhang}@ynu.edu.cn Abstract Toxic span detection requires the detection of spans that make a text toxic instead of simply classifying the text. In this paper, a transformer-based model with auxiliary infor- 5.mation is proposed for SemEval-2021 Task The proposed model was implemented based on the BERT-CRF architecture. It con- sists of three parts: a transformer-based model that can obtain the token representation, an auxiliary information module that combines features from different layers, and an out- put layer used for the classification. Vari- ous BERT-based models, such as BERT, AL- BERT, RoBERTa, and XLNET, were used to learn contextual representations. The predic- tions of these models were assembled to im- prove the sequence labeling tasks by using a voting strategy. Experimental results showed that the introduced auxiliary information can improve the performance of toxic spans detec- tion. The proposed model ranked 5th of 91 in the competition. The code of this study is avail- able at https://github.com/Chenrj233/ semeval2021_task5 1 Introduction Existing toxicity detection datasets and models classify the entire comment or document and do not identify the range that makes the text toxic. A system that accurately locates the toxicity range in the text is crucial in achieving semi-automatic re- view. As a complete submission for the shared task, systems are required to extract a list of toxic spans or an empty list per text. We define a sequence of words that attribute to the text’s toxicity as the toxic span. Table 1 shows two toxic spans, ”stupid” and ”a!@#!@,” which have character offsets from 10 to 15 (counting starts from 0) and from 51 to 56, respectively. Systems are then expected to return the offset list for this text. Text This is a stupid example, so thank you for nothing a!@#!@. Offset List [10,11,12,13,14,15,51,52,53,54,55,56] Table 1: Example of toxic spans detection shared task. The main purpose of this task is to identify the toxic spans in a given text; this can be transformed into a sequence labeling task in natural language processing. Unlike normal sequence labeling tasks, this task is more challenging because the toxic spans in the text may involve a word, phrase, or even a sentence. Traditional methods used to ad- dress the problem of sequence labeling include con- ditional random fields (CRF) (Lafferty et al., 1999), combined models of both long-short-term mem- ory and CRF (LSTM-CRF) (Gupta et al., 2019), and bidirectional encoder representation from trans- formers (BERT) (Devlin et al., 2019). In this study, we use BERT, ALBERT (Lan et al., 2019), RoBERTa (Liu et al., 2019), and XLNET (Yang et al., 2019) to solve this problem. Com- pared with the conventional model, our model adds auxiliary …

Content truncated to 500 words