Pre-training and Evaluating Transformer-based Language Models for Icelandic

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 753–762 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 753 Criteria for the Annotation of Implicit Stereotypes Wolfgang S. Schmeisser-Nieto1,2, Montserrat Nofre1, Mariona Taul´e1,2 1. Universitat de Barcelona-CLiC, Centre de Llenguatge i Computaci´o 2. UBICS, Universitat de Barcelona-Institute of Complex Systems Gran Via de les Corts Catalanes, 585 - 08007, Barcelona {wolfgang.schmeisser, montsenofre, mtaule}@ub.edu Abstract The growth of social media has brought with it a massive channel for spreading and reinforcing stereotypes. This issue becomes critical when the affected targets are minority groups such as women, the LGBT+ community and immigrants. Although from the perspective of computational linguistics, the detection of this kind of stereotypes is steadily improving, most stereotypes are expressed implicitly and identifying them automatically remains a challenge. One of the problems we found for tackling this issue is the lack of an operationalised definition of implicit stereotypes that would allow us to annotate consistently new corpora by characterising the different forms in which stereotypes appear. In this paper, we present thirteen criteria for annotating implicitness which were elaborated to facilitate the subjective task of identifying the presence of stereotypes. We also present NewsCom-Implicitness, a corpus of 1,911 sentences, of which 426 comprise explicit and implicit racial stereotypes. An experiment was carried out to evaluate the applicability of these criteria. The results indicate that different criteria obtain different inter-annotator agreement values and that there is a greater agreement when more criteria can be identified in one sentence. Keywords: Stereotype, Implicitness, Annotation 1. Introduction Over the past few years, the detection and classifica- tion of stereotypes and biased speech have gained at- tention within NLP, coinciding with the rise of social media. The web user has become not only a consumer of content but also a generator, which has facilitated the spread of toxic and hate speech, especially when it is directed towards minority groups. Toxic language typically contains stereotypes, so in order to tackle this issue in a thorough way, there exists the need to identify them automatically in the different ways they are ex- pressed. From this perspective, we distinguish explicit stereotypes, which are easy to recognise, from implicit stereotypes, which are indirectly conveyed within the message and are sometimes even harmless in appear- ance. The process of inference required for humans to interpret a stereotype can therefore be complex, and stereotypes need to be previously well defined for them to be automatically detected and classified successfully. In this work, due to the complexity and subjectivity, as well as the lack of a clear conceptualisation of the task involved in identifying implicitly expressed stereo- types, we propose criteria that will serve as guidelines for the annotation of the NewsCom-Implicitness cor- pus, a subset of the NewsCom-TOX corpus (Taul´e et al., 2021). After a thorough observation of carefully selected data consisting of user comments on news arti- cles related to immigration, we extracted linguistic pat- terns, mostly at the discursive …

Content truncated to 500 words