University at Buffalo at SemEval-2023 Task 11: MASDA–Modelling Annotator Sensibilities through DisAggregation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 978–985 July 13-14, 2023 ©2023 Association for Computational Linguistics University at Buffalo at SemEval-2023 Task 11: MASDA–Modelling Annotator Sensibilities through DisAggregation Michael J. Sullivan† Mohammed Nasheed Yasin† Cassandra L. Jacobs Department of Linguistics University at Buffalo {mjs227;m44;cxjacobs}@buffalo.edu † denotes equal contribution Abstract Modeling the most likely label when an anno- tation task is perspective-dependent discards relevant sources of variation that come from the annotators themselves. We present three ap- proaches to modeling the controversiality of a particular text. First, we explicitly represented annotators using annotator embeddings to pre- dict the training signals of each annotator’s se- lections in addition to a majority class label. This method leads to reduction in error rela- tive to models without these features, allow- ing the overall result to influence the weights of each annotator on the final prediction. In a second set of experiments, annotators were not modeled individually but instead annotator judgments were combined in a pairwise fashion that allowed us to implicitly combine annota- tors. Overall, we found that aggregating and explicitly comparing annotators’ responses to a static document representation produced high- quality predictions in all datasets, though some systems struggle to account for large or variable numbers of annotators. 1 Introduction The primary challenge of accommodating disagree- ment in natural language processing is in aggregat- ing (or disaggregating) the decisions of individual annotators. As part of this submission to LeWiDi23 (Leonardellli et al., 2023), we present several pos- sible approaches, deployed to varying degrees of success, for modeling annotators individually or in parallel. Building on the general idea that individ- ual annotators largely process language in similar ways, minus differences of political opinion or life experience, we create features that allow us to com- pare and contrast annotators to each other directly. Our first approach creates annotator layers that represent individual annotators using small- dimensionality embeddings. The decisions of the annotators (their personal label) then impact the final (gold) label. These separate networks enable the joint prediction of each annotator while con- sidering the sentence embedding separately. We present two ablation studies showing that integrat- ing the features in this way is critical for perfor- mance of the system. 2 Approach 1: Annotator Layers and Document Embeddings The model consists of a pretrained sentence trans- former coupled with a classification head (see Fig- ure 1). We use this template for the HS-Brexit (Akhtar et al., 2021) and ArMIS (Almanea and Poesio, 2022) datasets, each modified to account for the main differences in the number of annota- tors and the language —we refer interested readers to the Appendix for dataset-specific modifications to this architecture. The model predicts a soft label for a given input, which is defined as the average of the predicted labels for each individual annotator (i.e. ensemble averaging; Dietterich, 2000). Aach annotator for the dataset is represented by a unique linear layer (AnnNet), which produces an embedding from the sentence transformer that reflects that annotator’s likely decision …

Content truncated to 500 words