AStarTwice at SemEval-2021 Task 5: Toxic Span Detection Using RoBERTa-CRF, Domain Specific Pre-Training and Self-Training

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 875–880 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 875 AStarTwice at SemEval-2021 Task 5: Toxic Span Detection using RoBERTa-CRF, Domain Specific Pre-Training and Self-Training Thakur Ashutosh Suman∗ Abhinav Jain∗ Indian Institute of Technology (Indian School of Mines) Dhanbad, India {ashutoshsuman99,jain.abhinav02}@gmail.com Abstract This paper describes our contribution to SemEval-2021 Task 5: Toxic Spans Detection. Our solution is built upon RoBERTa language model and Conditional Random Fields (CRF). We pre-trained RoBERTa on Civil Comments dataset, enabling it to create better contextual representation for this task. We also employed the semi-supervised learning technique of self- training, which allowed us to extend our train- ing dataset. In addition to these, we also iden- tified some pre-processing steps that signifi- cantly improved our F1 score. Our proposed system achieved a rank of 41 with an F1 score of 66.16%. 1 Introduction In recent years there has been an exponential in- crease in the use of social network platforms. With rising abusive language and hate on such plat- forms, it is more important than ever to main- tain online conversations constructive and inclu- sive. This problem can be tackled by filtering toxic comments/posts. The massive volume of data gen- erated at a fast pace makes manually filtering each comment complicated and time-consuming. This process can be automated by modelling it as a su- pervised classification problem. A similar task was proposed in SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) (Zampieri et al., 2019). Most of the top-ranked teams in this task used transformer lan- guage models (Liu et al., 2019a; Zhu et al., 2019; Pelicon et al., 2019; Wu et al., 2019) or an ensemble of CNN and RNN (Mahata et al., 2019; Mitrovi´c et al., 2019) to classify the sentences. The problem with the above approach is that it doesn’t give moderators much knowledge about the reason for a sentence’s toxicity. Highlighting ∗Equal Contribution. Author order determined by a coin flip toxic spans can help human moderators who fre- quently deal with long comments and prefer at- tribution rather than just an unexplained toxicity score. SemEval 2021 Task 5: Toxic Span Detection (Pavlopoulos et al., 2021) gave a chance to propose NLP systems to solve this problem. The task is con- cerned with developing systems that can recognise spans that contribute to the text’s toxicity. This task had a few challenges. Since the sam- ples were from an online commenting platform, they were grammatically incorrect and consisted of many out of vocabulary words. The noisy and am- biguous structure of comments significantly ham- pers the performance of general NLP models. The training dataset had a little less than 8000 sam- ples. Thus, there was a need to select systems that can produce meaningful results, even with a lim- ited number of training samples. Undoubtedly, the hardest part is to identify spans that can account for the toxicity of the sample. The span could be as …

Content truncated to 500 words