Unifying Data Perspectivism and Personalization: An Application to Social Norms

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7391 - 7402 December 7-11, 2022 ©2022 Association for Computational Linguistics Unifying Data Perspectivism and Personalization: An Application to Social Norms Joan Plepi † and Béla Neuendorf † and Lucie Flek †‡ and Charles Welch †‡ † Conversational AI and Social Analytics (CAISA) Lab Department of Mathematics and Computer Science, University of Marburg ‡ The Hessian Center for Artificial Intelligence (Hessian.AI) {plepi,neuendob,lucie.flek,welchc}@uni-marburg.de Abstract Instead of using a single ground truth for lan- guage processing tasks, several recent studies have examined how to represent and predict the labels of the set of annotators. However, often little or no information about annotators is known, or the set of annotators is small. In this work, we examine a corpus of social media posts about conflict from a set of 13k anno- tators and 210k judgements of social norms. We provide a novel experimental setup that ap- plies personalization methods to the modeling of annotators and compare their effectiveness for predicting the perception of social norms. We further provide an analysis of performance across subsets of social situations that vary by the closeness of the relationship between par- ties in conflict, and assess where personaliza- tion helps the most. 1 Introduction Obtaining a single ground truth is not possible or necessary for subjective natural language classifica- tion tasks (Ovesdotter Alm, 2011). Each annotator is a person with their own feelings, thoughts, expe- riences, and perspectives (Basile et al., 2021b). In fact, researchers have been calling for the release of data without an aggregated ground truth, and for evaluation that takes individual perspectives into account (Flek, 2020). The idea that each annotator has their own view of subjective tasks, and even those previously thought to be objective was introduced by Basile (2020) as data perspectivism. A growth in the in- terest of this viewpoint has led to the 1st Workshop on Perspectivist Approaches to NLP in 2022. Work has examined how to model annotators for subjec- tive tasks and to predict each annotator’s label (Da- vani et al., 2021; Fornaciari et al., 2021). Model- ing annotator perspectives requires the release of corpora that include annotator-level labels rather than aggregated “ground truth” labels. Bender and Friedman (2018) further recommend releasing data statements that describe characteristics including who is represented in the data and the demograph- ics of annotators. Such information is beneficial for raising awareness of the biases in our data. While some corpora contain this information, like those for humor, emotion recognition, and hateful or offensive language, they contain few annotators and no additional information about them (Meaney et al., 2021; Kennedy et al., 2018; Demszky et al., 2020). An additional complication for subjective tasks is the fact that different people will interpret text in different ways. What is deemed toxic or offen- sive depends on who you ask (Sap et al., 2021; Leonardelli et al., 2021). There are notable differ- ences in perceived and intended sarcasm (Oprea and Magdy, 2019; Plepi …

Content truncated to 500 words