TeamAmpa at SemEval-2023 Task 3: Exploring Multilabel and Multilingual RoBERTa Models for Persuasion and Framing Detection

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 847–855 July 13-14, 2023 ©2023 Association for Computational Linguistics TeamAmpa at SemEval-2023 Task 3: Exploring Multilabel and Multilingual RoBERTa Models for Persuasion and Framing Detection Amalie Brogaard Pauli1 Rafael Pablos Sarabia1 Leon Derczynski2 Ira Assent1 1Department of Computer Science, Aarhus University, Denmark 2IT University of Copenhagen, Denmark {ampa,rpablos,ira}@cs.au.dk, ld@itu.dk Abstract This paper describes our submission to the SemEval 2023 Task 3 on two subtasks: detect- ing persuasion techniques and framing. Both subtasks are multi-label classification problems. We present a set of experiments, exploring how to get robust performance across languages us- ing pre-trained RoBERTa models. We test dif- ferent oversampling strategies, a strategy of adding textual features from predictions ob- tained with related models, and present both inconclusive and negative results. We achieve a robust ranking across languages and subtasks with our best ranking being nr. 1 for Subtask 3 on Spanish. 1 Introduction Task 3 at SemEval-2023 (Piskorski et al., 2023) promotes research on propaganda detection by pro- viding a shared task with annotated data for detect- ing framing (Subtask 2) and persuasion techniques (Subtask 3) in online news from various languages; namely, English, French, German, Italian, Polish, and Russian. In addition, three additional surprise languages were included in the test phase: Spanish, Greek, and Georgian. We have submitted predictions for each of the nine languages on both Subtask 2 and Subtask 3, with both subtasks being multi-label classification problems. We obtain robust results across the dif- ferent languages by fine-tuning RoBERTa models (Liu et al., 2019). Our best rankings on Subtask 3 are nr. 1 on Spanish and nr. 2 on French and Russian. On Subtask 2, we rank nr. 2 on English and Greek. In our paper, we study how to get good per- formance on multi-label classification using fine- tuning of transformer-based models on both sub- tasks and across languages. More concretely, we conduct experiments and report results on the fol- lowing three questions. First, we explore when to use an English backbone with less annotated data and when to use a multilingual backbone to exploit a larger set of multilingual annotated data. Second, we study different oversampling strategies. Third, we try to directly include textual features in training for persuasion detection of the broader categories of the fallacy of ethos, pathos and logos. We report both positive and negative results. We made our final training scripts available on GitHub1. 2 Background 2.1 Related work Detecting persuasion techniques has recently got- ten increased attention in the community; It has been addressed in different forms in former Shared Tasks (Martino et al., 2020; Dimitrov et al., 2021). But also in related work e.g. detecting logical fal- lacies (Habernal et al., 2017; Jin et al., 2022) and personal attacks in debates/communication (Haber- nal et al., 2018; Sheng et al., 2021). The study of Pauli et al. (2022) tries to reframe different cate- gories from such previous studies into the fallacy of ethos, pathos and logos. In …

Content truncated to 500 words