FTD at SemEval-2023 Task 3: News Genre and Propaganda Detection by Comparing Mono- and Multilingual Models with Fine-tuning on Additional Data

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 549–555 July 13-14, 2023 ©2023 Association for Computational Linguistics FTD at SemEval-2023 Task 3: News Genre and Propaganda Detection by Comparing Mono- and Multilingual Models with Fine-tuning on Additional Data Mikhail Lepekhin MIPT / Moscow lepehin.mn@phystech.edu Serge Sharoff University of Leeds / UK s.sharoff@leeds.ac.uk Abstract We report our participation in the SemEval- 2023 shared task on propaganda detection and describe our solutions with pre-trained mod- els and their ensembles. For Subtask 1 (News Genre Categorisation), we report the impact of several settings, such as the choice of the clas- sification models (monolingual or multilingual or their ensembles), the choice of the training sets (base or additional sources), the impact of detection certainty in making a classification decision as well as the impact of other hyper- parameters. In particular, we fine-tune models on additional data for other genre classifica- tion tasks, such as FTD. We also try adding texts from genre-homogenous corpora, such as Panorama, Babylon Bee for satire and Gi- ganews for for reporting texts. We also make prepared models for Subtasks 2 and 3 with fine- tuning the corresponding models first for Sub- task 1. The code needed to reproduce the ex- periments is available.1 1 Introduction Non-topical text classification includes a wide range of tasks aimed at predicting a text property that is not connected directly to a text topic, so it is not simply detected via keywords, for example, predicting a text style, politeness, difficulty level, the age or the first language of its author, etc. Au- tomatic genre identification (Santini et al., 2010) is one of the standard problems of non-topical text classification. It can be applied in many areas such as information retrieval, language teaching or lin- guistic research to find texts of specific genres for any topic. SemEval Shared Task 3 consists of 3 subtasks linked to the concept of non-topical text classification. Subtask 1 (News Genre Categorisa- tion) is aimed at predicting expression of opinions, objective news reporting, or satire on the text level. Subtask 2 is on prediction of text framing with the 1https://github.com/MikeLepekhin/SemEval2023_ task3 - our GitHub repository possibility of multiple labels per article. Subtask 3 is about detecting the kind of persuation techniques on the paragraph level. NLP shifted dramatically when large pre-trained models emerged. Multilingual BERT (Pires et al., 2019) shows impressive results on a variety of lan- guages and often it even manages to beat the mono- lingual models. XLM-RoBERTa (Conneau et al., 2019) is a more powerful multilingual model. It is trained on more than 100 languages and uses a vocabulary with a higher number of tokens - more than 200K. A problem with many non-topical classification setups concerns robustness. When a classifier sees a text that differs drastically from any text in the train set, it tends to misclassify it. And quite often, it is accompanied by a low probability of the max probable class. A recent study by (Tsymbalov et al., 2020) represents …

Content truncated to 500 words