UNTLing at SemEval-2020 Task 11: Detection of Propaganda Techniques in English News Articles

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1847–1852 Barcelona, Spain (Online), December 12, 2020. 1847 UNTLing at SemEval-2020 Task 11: Detection of Propaganda Techniques in English News Articles Maia Petee and Alexis Palmer The University of North Texas Denton, Texas maiapetee@my.unt.edu, alexis.palmer@unt.edu Abstract Our system for the PropEval task explores the ability of syntactic and semantic features to detect and label propagandistic rhetorical techniques in English news articles. For Subtask 2, labeling identified propagandistic fragments with one of fourteen technique labels, our system attains a micro-averaged F1 of 0.40; in this paper, we take a detailed look at the fourteen labels and how well our model detects each of them. We also propose strategies to fill the gaps. 1 Introduction This paper describes the UNT Linguistics contribution to the Propaganda Evaluation (PropEval) shared task at SemEval-2020 (Da San Martino et al., 2020). Our system uses non-neural classification methods to address the two linked subtasks: 1) detection of variable-length English-language news article spans that employ propagandistic rhetorical techniques; and 2) 14-way labeling of identified spans. The system focuses on semantically-oriented features, including word embeddings, sentiment/valence analysis, and named entity features. The amount of text available in our vast global landscape, both content created by companies for public use and user-generated content, has grown far beyond the reach of human moderation. The prevalence of content with misleading rhetorical techniques, often promoting partisan political or social agendas, calls for automated detection and labeling of propaganda in textual media. Currently, systems like this are rare (for example, Proppy (Da San Martino et al., 2019)), and to our knowledge there are none in wide use. Widely-available, browser-based, automated, interactive detection of online propaganda could reduce the quantity of that propaganda and promote the formation of a better-informed public, acting as a bulwark against misleading content in real time. Such systems could influence rhetorical trends in the presentation of online news to skew away from the unnecessarily sensational and toward the factual. Our system approaches the task of propaganda detection primarily from a semantic perspective, following the intuition that propagandistic text is different from non-propagandistic text in ways that are linguistically nuanced. The semantic information encoded in word and sentence embeddings, for example, has been shown to capture such nuances (Chen et al., 2013). We focus primarily on the Subtask 2 (Technique Classification) and made an official submission for this task only. Our Subtask 1 system achieved a macro-averaged F1 score of 0.349, but only after the end of the official submission period. Our system for Subtask 2 came in 27th out of 32 teams on the test set (F = 0.391) and 34th out of 42 teams on the development set (F = 0.409). Detailed results and error analysis appear below. 2 Data and System Overview Data. The task data are English-language news articles scraped from the web. Within each article, gold standard propagandistic fragments are indicated via character offsets and then labeled with one of the 14 technique labels. In all, …

Content truncated to 500 words