SemEval-2023 Task 10: Explainable Detection of Online Sexism

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2193–2210 July 13-14, 2023 ©2023 Association for Computational Linguistics SemEval-2023 Task 10: Explainable Detection of Online Sexism Hannah Rose Kirk1,2*, Wenjie Yin1,3*, Bertie Vidgen1, and Paul Röttger1,2 1Rewire 2University of Oxford 3Queen Mary University of London Abstract Online sexism is a widespread and harmful phenomenon. Automated tools can assist the detection of sexism at scale. Binary detection, however, disregards the diversity of sexist con- tent, and fails to provide clear explanations for why something is sexist. To address this issue, we introduce SemEval Task 10 on the Explain- able Detection of Online Sexism (EDOS). We make three main contributions: i) a novel hi- erarchical taxonomy of sexist content, which includes granular vectors of sexism to aid ex- plainability; ii) a new dataset of 20,000 so- cial media comments with fine-grained labels, along with larger unlabelled datasets for model adaptation; and iii) baseline models as well as an analysis of the methods, results and errors for participant submissions to our task. Content warning: We show illustrative examples of sexist language to describe the taxonomy and analyse error types. 1 Introduction Online sexism can inflict harm on women who are targeted, make online spaces inaccessible and un- welcoming, and perpetuate social asymmetries and injustices. Automated tools are now widely de- ployed to find sexist content at scale, supporting content moderation, monitoring and research. How- ever, most automated classifiers do not give further explanations beyond generic, high-level categories such as ‘toxicity’, ‘abuse’ or ‘sexism’. Opaque classification without explanations can cause con- fusion, anger and mistrust among moderated users, and make it harder to challenge moderation mis- takes; moderators who use automated tools may not trust or fully understand the labels, reducing their efficiency; and it is difficult to assess model weaknesses in the absence of granular labels, which hinders the development of better models. We organised the SemEval 2023 Task Explain- able Detection of Online Sexism (EDOS), which *Equal contribution follows from success of previous shared tasks on abuse and hate detection (Zampieri et al., 2019; Ribeiro and Silva, 2019; Pavlopoulos et al., 2021; Basile et al., 2019; Fersini et al., 2018a,b). Our task proposes and applies a taxonomy with three hierar- chical tasks for detecting sexist content. Task A is a binary task to detect whether content is sexist. For sexist content, Task B distinguishes between four distinct categories of sexism, and Task C identifies one of 11 fine-grained sexism vectors. There are three factors that make our EDOS task unique: First, data diversity: we sample data from two large social media platforms (Reddit and Gab) using an ensemble of filtering methods. The entries are more diverse than sampling from keywords alone or a single platform. Second, annotation quality: we recruit highly-trained annotators who all self-identify as women. We make this deci- sion to mitigate implicit biases in labelling and to evoke a participatory approach to AI where the communities primarily harmed from specific forms of content are included in …

Content truncated to 500 words