SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1377–1414 Barcelona, Spain (Online), December 12, 2020. 1377 SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles Giovanni Da San Martino1, Alberto Barr´on-Cede˜no2, Henning Wachsmuth3, Rostislav Petrov4 and Preslav Nakov1 1Qatar Computing Research Institute, HBKU, Qatar 2Universit`a di Bologna, Forl`ı, Italy 3Paderborn University, Paderborn, Germany 4A Data Pro, Sofia, Bulgaria {gmartino, pnakov}@hbku.edu.qa a.barron@unibo.it henningw@upb.de rostislav.petrov@adata.pro Abstract We present the results and the main findings of SemEval-2020 Task 11 on Detection of Propa- ganda Techniques in News Articles. The task featured two subtasks. Subtask SI is about Span Identification: given a plain-text document, spot the specific text fragments containing propaganda. Subtask TC is about Technique Classification: given a specific text fragment, in the context of a full document, determine the propaganda technique it uses, choosing from an inventory of 14 possible propaganda techniques. The task attracted a large number of participants: 250 teams signed up to participate and 44 made a submission on the test set. In this paper, we present the task, analyze the results, and discuss the system submissions and the methods they used. For both subtasks, the best systems used pre-trained Transformers and ensembles. 1 Introduction Propaganda aims at influencing people’s mindset with the purpose of advancing a specific agenda. It can hide in news published by both established and non-established outlets, and, in the Internet era, it has the potential of reaching very large audiences (Muller, 2018; Tard´aguila et al., 2018; Glowacki et al., 2018). Propaganda is most successful when it goes unnoticed by the reader, and it often takes some training for people to be able to spot it. The task is way more difficult for inexperienced users, and the volume of text produced on a daily basis makes it difficult for experts to cope with it manually. With the recent interest in “fake news”, the detection of propaganda or highly biased texts has emerged as an active research area. However, most previous work has performed analysis at the document level only (Rashkin et al., 2017; Barr´on-Cede˜no et al., 2019a) or has analyzed the general patterns of online propaganda (Garimella et al., 2015; Chatfield et al., 2015). SemEval-2020 Task 11 offers a different perspective: a fine-grained analysis of the text that comple- ments existing approaches and can, in principle, be combined with them. Propaganda in text (and in other channels) is conveyed through the use of diverse propaganda techniques (Miller, 1939), which range from leveraging on the emotions of the audience —such as using loaded language or appeals to fear— to using logical fallacies —such as straw men (misrepresenting someone’s opinion), hidden ad-hominem fallacies, and red herring (presenting irrelevant data). Some of these techniques have been studied in tasks such as hate speech detection (Gao et al., 2017) and computational argumentation (Habernal et al., 2018). Figure 1 shows the fine-grained propaganda identification pipeline, including the two targeted subtasks. Our goal is to facilitate the development of models capable of spotting text fragments where propaganda techniques …

Content truncated to 500 words