ULFRI at SemEval-2022 Task 4: Leveraging uncertainty and additional knowledge for patronizing and condescending language detection

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 525 - 532 July 14-15, 2022 ©2022 Association for Computational Linguistics ULFRI at SemEval-2022 Task 4: Leveraging uncertainty and additional knowledge for patronizing and condescending language detection Matej Klemen Marko Robnik-Šikonja University of Ljubljana, Faculty of Computer and Information Science Veˇcna pot 113, Ljubljana, Slovenia {matej.klemen, marko.robnik}@fri.uni-lj.si Abstract We describe the ULFRI system used in the Sub- task 1 of SemEval-2022 Task 4 Patronizing and condescending language detection. Our models are based on the RoBERTa model, modified in two ways: (1) by injecting additional knowl- edge (coreferences, named entities, dependency relations, and sentiment) and (2) by leveraging the task uncertainty by using soft labels, Monte Carlo dropout, and threshold optimization. We find that the injection of additional knowledge is not helpful but the uncertainty management mechanisms lead to small but consistent im- provements. Our final system based on these findings achieves F1 = 0.575 in the online evaluation, ranking 19th out of 78 systems. 1 Introduction Despite invaluable contributions to the society, the internet can also serve as an infrastructure for a rapid spread of hurtful language, in part due to the anonymity it commonly provides (Burnap and Williams, 2015). The spread of such language can have a serious impact on individuals, such as the increased development of mental health problems in children (Munro, 2011). To prevent this, the society has to establish moderation mechanisms. Fully manual content moderation is infeasible both due to the large scale of the web as well as the possible negative psychological effects on human moderators (Arsht and Etcovitch, 2018). Much at- tention has been devoted to the automatic detection of offensive language within the field of natural lan- guage processing (NLP). Some examples include the detection of hate speech (Davidson et al., 2017), toxic language (Pavlopoulos et al., 2021), and cy- berbullying (Dadvar et al., 2013), which use a rela- tively explicit form of hurtful language (Waseem et al., 2017). In contrast, patronizing and conde- scending language (PCL) is more implicit in nature. PCL can roughly be described as an expression of a superior attitude towards others, possibly uncon- sciously. Perez Almendros et al. (2020) have shown that large language models are able to detect PCL to a various degree, but consistently better than random guessing or a machine learning approach using the bag-of-words representation. Based on that, the authors propose SemEval-2022 Task 4 (Pérez-Almendros et al., 2022), which aims to en- courage further research and improvements in the detection of PCL. We present our attempts at modeling PCL, based on the RoBERTa model (Liu et al., 2019) and fol- lowing two main lines: 1. Injection of additional knowledge. We experiment with the injection of additional knowledge on coreferences, named entities, dependency relations, and sentiment. 2. Leveraging uncertainty present in the task. We experiment with the use of soft labels in the form of the target label probability distri- bution, and with the Monte Carlo dropout as a means for more accurate estimation …

Content truncated to 500 words