NUIG at SemEval-2020 Task 12: Pseudo Labelling for Offensive Content Classification

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1598–1604 Barcelona, Spain (Online), December 12, 2020. 1598 NUIG at SemEval-2020 Task 12: Pseudo labelling for offensive content classification Shardul Suryawanshi, Mihael Arcan, Paul Buitelaar Insight SFI Research Centre for Data Analytics Data Science Institute, National University of Ireland Galway shardul.suryawanshi@insight-centre.org Abstract This work addresses the classification problem defined by sub-task A (English only) of the OffensEval 2020 challenge. We used a semi-supervised approach to classify given tweets into an offensive (OFF) or not-offensive (NOT) class. As the OffensEval 2020 dataset is loosely labelled with confidence scores given by unsupervised models, we used last year’s offensive language identification dataset (OLID) to label the OffensEval 2020 dataset. Our approach uses a pseudo-labelling method to annotate the current dataset. We trained four text classifiers on the OLID dataset and the classifier with the highest macro-averaged F1-score has been used to pseudo label the OffensEval 2020 dataset. The same model which performed best amongst four text classifiers on OLID dataset has been trained on the combined dataset of OLID and pseudo labelled OffensEval 2020. We evaluated the classifiers with precision, recall and macro-averaged F1-score as the primary evaluation metric on the OLID and OffensEval 2020 datasets. 1 Introduction We are generating data at a rate which has never been seen before1, while social media plays a big role in our everyday life. Most people with smartphones have a social media account and an active social media life. However, due to anonymity and freedom of expression, social media has become a breeding ground for offensive content. This content can be deemed as offensive if it intends to demean a person or a group of people due to their ethnicity, sexual orientation or by a personal attack2. This raises the need for an evaluation system that could automatically detect such offensive text. Sub-task A (English only) in the OffensEval 2020 (Zampieri et al., 2020) provided the OffensEval 2020 dataset (Zampieri et al., 2019a) of English tweets, which has been labelled with average confidence and standard deviation of the confidence scores, e.g. “@USER His ass need to stay up” has an average confidence of 0.833496 while standard deviation is 0.140625. Rosenthal et al. (2020) used an unsupervised learning approach to produce these confidence scores. Since the system is evaluated on its label prediction performance (offensive OFF, not-offensive NOT) and not on these scores, it was up to the participant to decide how to best map the scores to labels. We trained different text classifiers on the OLID (Zampieri et al., 2019a) dataset to predict the labels of the OffensEval 2020. This pseudo-labelling approach avoids the efforts involved in the manual adjustment of threshold values of average confidence (Avg conf) and average standard deviation (Avg std). 2 Related work There has been significant research on aggression detection, hate speech detection (Ranjan et al., 2016; Rani et al., 2020; Jose et al., 2020) and cyberbullying (Schmidt and Wiegand, 2017). Based on this This work is licensed under a …

Content truncated to 500 words