APatt at SemEval-2023 Task 3: The Sapienza NLP System for Ensemble-based Multilingual Propaganda Detection

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 382–388 July 13-14, 2023 ©2023 Association for Computational Linguistics APatt at SemEval-2023 Task 3: The Sapienza NLP System for Ensemble-based Multilingual Propaganda Detection Antonio Purificato and Roberto Navigli Department of Computer, Control and Management Engineering Sapienza University of Rome purificato.2019135@studenti.uniroma1.it, navigli@diag.uniroma1.it Abstract In this paper, we present our approach to the task of identification of persuasion techniques in text, which is a subtask of the SemEval- 2023 Task 3 on the multilingual detection of genre, framing, and persuasion techniques in online news. The subtask is multi-label at the paragraph level and the inventory considered by the organizers covers 23 persuasion tech- niques. Our solution is based on an ensem- ble of a variety of pre-trained language models (PLMs) fine-tuned on the propaganda dataset. We first describe our system, the different ex- perimental setups we considered, and then pro- vide the results on the dev and test sets re- leased by the organizers. The official evalua- tion shows our solution ranks 1st in English and attains high scores in all the other languages, i.e. French, German, Italian, Polish, and Rus- sian. We also perform an extensive analysis of the data and the annotations to investigate how they can influence the quality of our systems. We release our code at https://github.com/ antoniopurificato/apatt_at_semeval. 1 Introduction Due to its diffusion on social media platforms, information is being shared in real-time in the modern digital world, in particular following the COVID-19 pandemic. The detection of propaganda has emerged as an important research topic with the recent interest in fake news (Da San Martino et al., 2020). In today’s information age, anyone can take advantage of the diffusion potential of Internet to spread propaganda surreptitiously, and this is in fact done by activist groups, businesses, religious institutions, the media, and even ordinary people, reaching vast audiences. Many different strategies are used to “hide” propaganda messages inside standard text, ranging from appealing to the audience’s emotions to utilizing logical fallacies. SemEval-2023 Task 3 (Piskorski et al., 2023) Subtask 3 offers a different way to investigate this problem: given a news article selected from a me- dia source that could potentially spread disinforma- tion, the system is tasked to identify the persuasion techniques adopted in each paragraph. This is a multi-label task at the paragraph level and also multilingual, with news articles in 6 different lan- guages: English, Italian, Polish, German, French and Russian. In our approach, we adopt transfer learning and, instead of using one single pre-trained language model, we create an ensemble of multiple state-of- the-art architectures. To output the final prediction two different approaches are used. For all the lan- guages except English it is possible to use a maxi- mum of two models, so for every ensemble we take the probabilities of these models and average them to make the final prediction. In English, instead, where it is possible to use more than two models for the ensemble, we do not average …

Content truncated to 500 words