MilaNLP at SemEval-2023 Task 10: Ensembling Domain-Adapted and Regularized Pretrained Language Models for Robust Sexism Detection

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2067–2074 July 13-14, 2023 ©2023 Association for Computational Linguistics MilaNLP at SemEval-2023 Task 10: Ensembling Domain-Adapted and Regularized Pretrained Language Models for Robust Sexism Detection Amanda Cercas Curry, Giuseppe Attanasio, Debora Nozza, Dirk Hovy Department of Computing Sciences Bocconi University, Milan, Italy {amanda.cercas,giuseppe.attanasio3,debora.nozza,dirk.hovy}@unibocconi.it Abstract We present the system proposed by the Mi- laNLP team for the Explainable Detection of Online Sexism (EDOS) shared task. We pro- pose an ensemble modeling approach to com- bine different classifiers trained with domain adaptation objectives and standard fine-tuning. Our results show that the ensemble is more robust than individual models and that regular- ized models generate more “conservative” pre- dictions, mitigating the effects of lexical over- fitting.1 However, our error analysis also finds that many of the misclassified instances are de- batable, raising questions about the objective annotatability of hate speech data. Warning: This paper contains examples of lan- guage that some people may find offensive. 1 Introduction Sexism and misogyny are ever-present in online spaces, creating a hostile environment which can prevent women from enjoying the benefits brought about by the internet (Jane, 2017). Because of its ubiquity, tackling sexism manually has proven in- effective, redirecting efforts towards automatic de- tection of this phenomenon using natural language processing. Kirk et al. (2023) presented a novel corpus for the classification of sexist content from Gab and Reddit in English using fine-grained labels. The hierarchical classification of sexism permits the de- velopment of classifiers that are both more accurate and explicable. In the Explainable Detection of Online Sexism (EDOS) shared task, systems are re- quired to detect sexist content. The task is divided into three sub-tasks that refer to different levels of granularity (see §2). In this paper, we propose an ensemble modeling approach to combine different fine-tuned language models. Although ensemble modeling has been 1Code available at https://github.com/MilaNLProc/ milanlp-at-edos. shown to be beneficial for various tasks in NLP (Garmash and Monz, 2016; Nozza et al., 2016; Fadel et al., 2019; Bashmal and AlZeer, 2021), its potential for hate speech detection has only been explored in a small number of papers (Zimmerman et al., 2018; Plaza-del Arco et al., 2019; Ramakrish- nan et al., 2019; Nozza, 2022). Our results confirm that ensemble learning improves robustness and classification performance and we find that regu- larized fine-tuning leads to higher uncertainty in sexism prediction. The resulting “conservative” models reduce both false positives and false nega- tives when included in the ensemble. Our system ranked 9th for the binary sexism detection task, with a macro F1-score of 0.8616 (-0.013 compared to the winning team). Moreover, through a deeper analysis on test in- stances, we find that many examples in the aggre- gated dataset are likely mislabelled – showing how to best annotate sexism (and other subjective lan- guage phenomena) remains an open question. 2 Task Overview The EDOS shared task (Kirk et al., 2023) requires participants to build and test automatic models for sexism detection for …

Content truncated to 500 words