MS@IW at SemEval-2022 Task 4: Patronising and Condescending Language Detection with Synthetically Generated Data

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 363 - 368 July 14-15, 2022 ©2022 Association for Computational Linguistics MS@IW at SemEval-2022 Task 4: Patronising and Condescending Language Detection with Synthetically Generated Data Selina Meyer, Maximilian Schmidhuber and Udo Kruschwitz Chair for Information Science University of Regensburg, Germany (selina.meyer, udo.kruschwitz)@ur.de maximilian.schmidhuber@stud.uni-regensburg.de Abstract In this description paper we outline the sys- tem architecture submitted to Task 4, Subtask 1 at SemEval-2022. We leverage the generative power of state-of-the-art generative pretrained transformer models to increase training set size and remedy class imbalance issues. Our best submitted system is trained on a synthetically enhanced dataset with 10.3 times as many posi- tive samples as the original dataset and reaches an F1 score of 50.62%, which is 10 percentage points higher than our initial system trained on an undersampled version of the original dataset. We explore possible reasons for the compara- bly low score in the overall task ranking and report on experiments conducted during the post-evaluation phase. 1 Introduction Task 4 of SemEval-2022 focuses on the detection of patronising and condescending language (PCL) in news (Pérez-Almendros et al., 2022). PCL in popular media and news sources is detrimental to an emancipated and equal society, as it is usually targeted towards minorities and socially disadvan- taged communities, often in an unsuccessful at- tempt to show solidarity (Perez Almendros et al., 2020). PCL has the potential to strengthen existing stereotypes by representing minorities either as pas- sive entities to be pitied and supported, thus taking away their agency and focusing on their vulnera- bilities or praising members of vulnerable groups for everyday achievements simply because of their background (Nolan and Mikami, 2013). In con- trast to hate speech, PCL is usually subtle, well intentioned, and free of discriminatory phrases or racial slurs, which makes it an interesting Natural Language Processing (NLP) problem. In other domains with more discriminatory classes such as hate speech detection, generative models have recently become increasingly popular and successful as a tool to increase classification performance (Wullach et al., 2021; Anaby-Tavor et al., 2020). In our contribution to the shared task, we explored to what extent this approach is feasible for the presented use case, where classification of a text sample is less distinct and often relies on world knowledge (Perez Almendros et al., 2020). The dataset provided for the task was fairly small, with less than 10% of the data belonging to the pos- itive class. We thus enhanced the original dataset in two ways for our system runs: • balancing the dataset by generating only PCL samples • increasing overall dataset size, by generating an equal amount of PCL and non-patronizing (nPCL) samples We generally followed the approach used by Wul- lach et al. (2021) and initially fine-tuned a BERT classifier on the original dataset. We then fine- tuned GPT-3 (Brown et al., 2020) and generated samples of PCL and nPCL which were classified using our fine-tuned system. Samples for which the BERT classification did not …

Content truncated to 500 words