SKAM at SemEval-2023 Task 10: Linguistic Feature Integration and Continuous Pretraining for Online Sexism Detection and Classification

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1805–1817 July 13-14, 2023 ©2023 Association for Computational Linguistics SKAM at SemEval-2023 Task 10: Linguistic Feature Integration and Continuous Pretraining for Online Sexism Detection and Classification Murali Manohar Kondragunta Amber Chen Karlo Slot Sanne Weering Tommaso Caselli University of Groningen {m.m.c.kondragunta, a.chen.1, k.h.r.slot, s.weering}@student.rug.nl t.caselli@rug.nl Abstract Sexism has been prevalent online. In this paper, we explored the effect of explicit linguistic fea- tures and continuous pretraining on the perfor- mance of pretrained language models in sexism detection. While adding linguistic features did not improve the performance of the model, con- tinuous pretraining did slightly boost the per- formance of the model in Task B from a mean macro-F11 score of 0.6156 to 0.6246. The best mean macro-F1 score in Task A was achieved by a finetuned HateBERT model using regu- lar pretraining (0.8331). We observed that the linguistic features did not improve the model’s performance. At the same time, continuous pretraining proved beneficial only for nuanced downstream tasks like Task-B. 1 Introduction Currently, a lot of models are proposed to flag on- line content containing sexism. However, these models only flag what is sexist content, but not explain why. Categorization of sexism will give a more detailed explanation as to why certain expres- sions are sexism. It leads to a better understanding of the decisions that the model makes, which can improve the model further. To overcome this limi- tation, SemEval 2023 Task 10 - Explainable Detec- tion of Online Sexism (Kirk et al., 2023), proposed three tasks, of which two will be discussed in this paper: • TASK A - Binary Sexism Detection: a two- class (or binary) classification problem where systems have to predict whether a post is sex- ist or not sexist. • TASK B - Category of Sexism: for posts that are sexist, a four-class classification problem 1Mean is calculated between local development set and co- dalab development set. Local development set is a 15 percent split of the training data provided. where systems have to predict one of four cat- egories: (1) threats, (2) derogation, (3) ani- mosity, (4) prejudiced discussions. In this paper, we aim to build a sexism detection system for Task A and B, which is built in two phases. First, we evaluate which language model performs best on the tasks. Later, we augment linguistic features to the best-performing language model in all possible combinations and benchmark them. Specifically, we consider BERT (Devlin et al., 2018a), HateBERT (Caselli et al., 2020) and De- BERTa (He et al., 2020). For linguistic features, we consider the ones described in Abburi et al. (2021): Empath, Hurtlex, and PerspectiveAPI. More de- tails on the models and features can be found in the appendix (Section A and B). From the phase 1 experiments, we observed HateBERT to perform better than BERT and De- BERTa on both tasks. This observation reinforces Gururangan et al. (2020)’s results on further pre- trained language models performing better on the …

Content truncated to 500 words