KInITVeraAI at SemEval-2023 Task 3: Simple yet Powerful Multilingual Fine-Tuning for Persuasion Techniques Detection

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 629–637 July 13-14, 2023 ©2023 Association for Computational Linguistics KInITVeraAI at SemEval-2023 Task 3: Simple yet Powerful Multilingual Fine-Tuning for Persuasion Techniques Detection Timo Hromadka1 and Timotej Smolen1 and Tomas Remis1 and Branislav Pecher2,1 and Ivan Srba1 1Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia 2Brno University of Technology, Brno, Czechia {timo.hromadka,timotej.smolen,tomas.remis}@intern.kinit.sk {branislav.pecher,ivan.srba}@kinit.sk Abstract This paper presents the best-performing solu- tion to the SemEval 2023 Task 3 on the sub- task 3 dedicated to persuasion techniques de- tection. Due to a high multilingual character of the input data and a large number of 23 pre- dicted labels (causing a lack of labelled data for some language-label combinations), we opted for fine-tuning pre-trained transformer- based language models. Conducting multi- ple experiments, we find the best configura- tion, which consists of large multilingual model (XLM-RoBERTa large) trained jointly on all input data, with carefully calibrated confidence thresholds for seen and surprise languages sep- arately. Our final system performed the best on 6 out of 9 languages (including two surprise languages) and achieved highly competitive re- sults on the remaining three languages. 1 Introduction The subtask 3 of the SemEval 2023 Task 3 aims at identifying persuasion techniques. The task is a multi-label one, where a model is required to iden- tify which of the 23 persuasion techniques (e.g., an appeal to authority) are present in a given para- graph. The paragraphs are obtained from articles in 6 languages (English, French, German, Italian, Polish, and Russian) collected between 2020 and mid 2022, revolving around widely discussed top- ics such as COVID-19, climate change, abortion, migration etc. Media sources are both mainstream and alternative news and web portals. Furthermore, the model is tested on 3 surprise languages (Greek, Georgian, and Spanish), for which labeled training data were not available. The importance of the task is eminent — automatically detected persuasion techniques can be utilized as credibility signals to assess content credibility and thus also to improve disinformation detection. The detailed description of the task is available in (Piskorski et al., 2023). In this paper, we propose a multilingual system, consisting of a single model tackling all languages. Our main strategy is to fine-tune a large pre-trained transformer-based language model. To find the best performing system, we experimented with differ- ent language models (and finally opted for XLM- RoBERTa large due to its performance), hyper- parameter tuning as well as confidence threshold calibration by changing the threshold for prediction in the multi-label classification. We also simulated the zero-shot setting on the training data to adjust the confidence threshold and better estimate the per- formance of our model on the surprise languages. Furthermore, we experiment with additional con- figurations, such as translating the data to a single language (English) and using it to fine-tune a mono- lingual model, applying various text pre-processing strategies, or layer freezing. However, these config- urations did not lead to improvements. Although our system is based on a rather sim- ple …

Content truncated to 500 words