UMass PCL at SemEval-2022 Task 4: Pre-trained Language Model Ensembles for Detecting Patronizing and Condescending Language

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 445 - 453 July 14-15, 2022 ©2022 Association for Computational Linguistics UMass PCL at SemEval-2022 Task 4: Pre-trained Language Model Ensembles for Detecting Patronizing and Condescending Language David Koleczek and Alex Scarlatos and Preshma Linet Pereira and Siddha Karakare University of Massachusetts Amherst {dkoleczek, ajscarlatos, ppereira, skarkare}@umass.edu Abstract Patronizing and condescending language (PCL) is everywhere, but rarely is the focus on its use by media towards vulnerable communities. Ac- curately detecting PCL of this form is a difficult task due to limited labeled data and how sub- tle it can be. In this paper, we describe our system for detecting such language which was submitted to SemEval 2022 Task 4: Patroniz- ing and Condescending Language Detection. Our approach uses an ensemble of pre-trained language models, data augmentation, and opti- mizing the threshold for detection. Experimen- tal results on the evaluation dataset released by the competition hosts show that our work is reliably able to detect PCL, achieving an F1 score of 55.47% on the binary classification task and a macro F1 score of 36.25% on the fine-grained, multi-label detection task. 1 Introduction Everyone has seen, experienced, and expressed patronizing and condescending language (PCL). Someone is patronizing or condescending when they communicate in a way that talks down to oth- ers, positions themselves in a superior position to the subjective group, or describes them in a char- itable way in order to raise a feeling of compas- sion for the target person or group. Such language is seen frequently in social media or other medi- ums where hate is the norm. Many previous works have addressed offensive language and hate speech (Zampieri et al., 2019) (Basile et al., 2019) in such communities. However, until (Perez-Almendros et al., 2020) there has not been work in model- ing PCL where 1) it is targeted towards vulnerable communities such as refugees or poor families 2) occurs in the general media or news. The authors note that this form of PCL is not usually meant to be harmful, and can often have good intentions, such as when the author is trying to raise awareness about an at-risk group. Yet, research in sociolin- guistics has shown that regardless of the intent, PCL has negative effects on potential vulnerable communities such as perpetuating stereotypes, rein- forcing societal misbehaviors, and oversimplifying deep-rooted problems. The authors of Don’t Pa- tronize Me! An Annotated Dataset with Patronizing and Condescending Language towards Vulnerable Communities originally proposed a new dataset to help in detecting such cases of PCL and are cur- rently hosting a SemEval-2022 competition: Pa- tronizing and Condescending Language Detection (Pérez-Almendros et al., 2022). In this paper we describe our contribution to both competition subtasks: binary and multi-label clas- sification. Our final ensemble leverages many tech- niques including pre-trained language models, data augmentation, intermediate fine-tuning on related datasets, and optimizing the detection threshold to empirically maximize F1. For the binary task, our system achieves an F1 score of 55.47%, …

Content truncated to 500 words