IREL at SemEval-2023 Task 11: User Conditioned Modelling for Toxicity Detection in Subjective Tasks

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2133–2136 July 13-14, 2023 ©2023 Association for Computational Linguistics IREL at SemEval-2023 Task 11: User Conditioned Modelling for Toxicity Detection in Subjective Tasks Ankita Maity, Pavan Kandru, Bhavyajeet Singh, Kancharla Aditya Hari and Vasudeva Varma IIIT Hyderabad {ankita.maity, siri.venkata, bhavyajeet.singh, aditya.hari}@research.iiit.ac.in vv@iiit.ac.in Abstract This paper describes our system used in the SemEval-2023 Task 11 Learning With Dis- agreements (Le-Wi-Di). This is a subjective task since it deals with detecting hate speech, misogyny and offensive language. Thus, dis- agreement among annotators is expected. We experiment with different settings like loss functions specific for subjective tasks and in- clude anonymized annotator-specific informa- tion to help us understand the level of disagree- ment. We perform an in-depth analysis of the performance discrepancy of these differ- ent modelling choices. Our system achieves a cross-entropy of 0.58, 4.01 and 3.70 on the test sets of HS-Brexit, ArMIS and MD-Agreement, respectively. Our code implementation is pub- licly available. 1 1 Introduction Natural language expressions, such as sentences and phrases, can often have multiple possible inter- pretations depending on the context in which they are used. This ambiguity arises due to language’s inherent complexity and flexibility, which can lead to different interpretations of the same expression by different individuals. Additionally, subjective tasks can lead to disagreements between annotators with different perspectives or interpretations of the same text. The SemEval-2023 task 11 Learning With Dis- agreements (Le-Wi-Di) (Leonardellli et al., 2023) focuses entirely on such subjective tasks, where training with aggregated labels makes much less sense. In this task, we worked with three (textual) datasets with different characteristics in terms of languages (English and Arabic), tasks (misogyny, hate speech, offensiveness detection) and annota- tions’ methodology (experts, specific demographic groups, AMT-crowd). We leverage this additional 1https://github.com/Ankita-Maity/LeWiDi information in order to get more accurate estimates of each annotator’s annotation. All the datasets provide a multiplicity of labels for each instance. The focus is on developing methods able to capture agreements/disagreements rather than focusing on developing the best model. Since a "truth" cannot be assumed, "soft" eval- uation is the primary form of evaluating perfor- mances, i.e. an evaluation that considers how well the model’s probabilities reflect the level of agree- ment among annotators. 2 Related Work This is the 2nd edition of the Le-Wi-Di task; the previous version was held in SemEval 2021 (Uma et al., 2021a). A survey paper by Uma et al. (2021b) was also released, which identified several NLP and CV tasks for which the gold-standard idealisation has been shown not to hold. It used them to anal- yse the extensive literature on learning from data possibly containing disagreements. Akhtar et al. (2021) who introduced the HS- Brexit dataset, trained different classifiers for each annotator, and then took an ensemble of classifiers. In the case of multiclass problems (for example, classifying different kinds of hate speech instead of simply distinguishing between hate speech vs non-hate speech), there have been efforts to frame it as which …

Content truncated to 500 words