DH-FBK at SemEval-2023 Task 10: Multi-Task Learning with Classifier Ensemble Agreement for Sexism Detection

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1894–1905 July 13-14, 2023 ©2023 Association for Computational Linguistics DH-FBK at SemEval-2023 Task 10: Multi-Task Learning with Classifier Ensemble Agreement for Sexism Detection Elisa Leonardelli Fondazione Bruno Kessler Trento, Italy eleonardelli@fbk.eu Camilla Casula Fondazione Bruno Kessler University of Trento Trento, Italy ccasula@fbk.eu Abstract This paper presents the submissions of the DH- FBK team for the three tasks of Task 10 at SemEval 2023. The Explainable Detection of Online Sexism (EDOS) task aims at detecting sexism in English text in an accurate and ex- plainable way, thanks to a fine-grained annota- tion that follows a three-level schema: sexist or not (Task A), category of sexism (Task B) and vector of sexism (Task C) exhibited. We use a multi-task learning approach in which models share representations from all three tasks, al- lowing for knowledge to be shared across them. Notably, with our approach a single model can solve all three tasks. In addition, motivated by the subjective nature of the task, we incorpo- rate inter-annotator agreement information in our multi-task architecture. Although disaggre- gated annotations are not available, we artifi- cially estimate them using a 5-classifier ensem- ble, and show that ensemble agreement can be a good approximation of crowd agreement. Our approach achieves competitive results, ranking 32nd out of 84, 24th out of 69 and 11th out of 63 for Tasks A, B and C respectively. We finally show that low inter-annotator agreement levels are associated with more challenging examples for models, making agreement information use- ful for this kind of task. � Warning: this paper contains exam- ples that may be offensive or upsetting. 1 Introduction The prevalence of sexism on the Internet can be damaging to the women it targets and make online spaces hostile. In order to address this, automated methods are often used to identify sexist content on a large scale, though typically only high-level cat- egories are identified, without further explanation. Improved interpretability and understanding of the decisions made by artificial intelligence when flag- ging what is sexist and why it is sexist could em- power both users and moderators. Recently, the SemEval-2023 Task 10 (Kirk et al., 2023) "Explain- able Detection of Online Sexism" (EDOS) has been proposed, to encourage research on sexist language detection in a more accurate and explainable way. This shared task focuses on exploring methods for identifying explicit sexism and classifying the type of behavior expressed into fine-grained categories in English data from Gab and Reddit. In this paper, we present the DH-FBK entries for the three tasks of EDOS. We propose a multi- task learning approach, a paradigm that leverages training signals of related tasks at the same time by exploiting a shared representation in the model. In particular, we simultaneously train models on all three tasks, corresponding to the three levels of the annotation schema utilized. The hypothesis is that the annotation levels contain mutually relevant information, making knowledge sharing beneficial for the execution of each task. …

Content truncated to 500 words